Graphics

 View Only
Back to discussions
Expand all | Collapse all

datalinq v11 folder with thousand .txt files

  • 1.  datalinq v11 folder with thousand .txt files

    Posted 20 days ago

    Hello,

    I am running XPression version 11 and I am running into an issue trying to configure a DataLinq source to read data from a large number of text files (around 6,500 files) located on a network share (UNC path).

    Context & Setup:

    • XPression Version: 11

    • File Location: \\nxtv-st1-fs01.rmc.local\DP1Share\

    • File Format: Plain text files using a hashtag (#) as a delimiter, with no headers (e.g., ACCOR-INTRADAY.txt).

    • Connection Type tried: ADODB (Microsoft.ACE.OLEDB.12.0) and Native Text DataLinq.

    The Problem:

    Inside XPression, I need to call these files dynamically based on a macro value (e.g., @id_society@-INTRADAY.txt or @id_society@-INTRADAY#txt).

    1. When using ADODB: I set up a schema.ini file in the folder to handle the # delimiter and formatting. However, DataLinq throws a path/file validation error. It completely rejects the UNC path (Failure: '\\...\' is not a valid path). If I map the network share to a drive letter (like V: or X:), XPression throws a "not a valid path" error as well, likely due to a Windows session/admin privilege conflict where the DataLinq service cannot see the mapped drive.

    2. When using Native Text DataLinq Source: Since the Text DataLinq Server configuration UI strictly requires selecting a specific physical file to validate the connection, I cannot leave it open to a directory. If I use a placeholder file in the server manager and then try to override it dynamically in the scene's DataLinq properties using the file=@id_society@-INTRADAY.txt syntax in the Table field, it fails to fetch the data.

    How can I properly configure XPression v11 to dynamically query thousands of individual text files on a network share using a specific delimiter, without hitting these path validation blocks?

    Any help or best practices for this workflow would be greatly appreciated!

    Thank you,



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------


  • 2.  RE: datalinq v11 folder with thousand .txt files

    Posted 20 days ago

    Must be a permissions issue. I was curious so I tried it and was able to get it to work fine.

    First, I setup an ADODB Datalinq as you described with this connection string: Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\\ls210d39a\share\Shared Documents\;Extended Properties="text;HDR=Yes;FMT=Delimited";

    And then created a schema.ini in the directory:

    [delete_me.csv]
Format=Delimited(#)
ColNameHeader=True
CharacterSet=ANSI

Col1=Name Text
Col2=Instrument Text

    And a data file for testing (delete_me.csv):

    Name#Instrument
Miles Davis#Trumpet
John Coltrane#Sax
Paul Chambers#Bass
Philly Joe Jones#Drums
Red Garland#Piano

    In the Xpression graphic, I created a text object for the file path. The text object is named "csv_file" and has a value of "delete_me#csv".

    Lastly, I create a text object to show the data with this datalinq:

    Column: Name
Row: 2
Table: @csv_file@

    And that text has the value of "John Coltrane".



    ------------------------------
    JohnCorigliano
    Senior Software Engineer
    ISC
    ------------------------------



  • 3.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Hi, thanks! I will try this today and keep you informed.



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 4.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Hi,

    I am having an issue with XPression DataLinq Manager when trying to configure a text file source.

    When I click on the "Browse" button, nothig append. Instead, the mouse cursor turns into a loading spinner, and the software hangs/frezes.

    What I have tried so far:

    • I checked Windows Registry permissions to ensure full access.

    • I made sure XPression has the necessary administrator rights.

    I have attached screenshots of my DataLinq configuration and the structure of the text file I am trying to read.

    Any help would be greatly appreciated. Thanks!



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 5.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Note that in my example, I am using the JET db engine and you are using ACE.

    Provider=Microsoft.Jet.OLEDB.4.0;Data Source=\\ls210d39a\share\Shared Documents\;Extended Properties="text;HDR=Yes;FMT=Delimited";

    This may be an issue because you have a 64-bit version of ACE installed whereas JET is 32 bit and so is DataLinq server. (I'm not 100% sure about this, but I believe it is correct.) What happens when you click "Test Connection"?

    If changing the engine doesn't work, try copying the files to a local folder and see it that works.



    ------------------------------
    JohnCorigliano
    Senior Software Engineer
    ISC
    ------------------------------

    Original Message


  • 6.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Hi,

    I have a "connection succesfully established" :) and my ACE is 32bit. i'll try with Jet. keep out in touch



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 7.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    same result with JET 4.0



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 8.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    same result on local drive :(



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 9.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    My broadcast department told me they are going to install version 12.5 this summer, so maybe it will work then.



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 10.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    This is a pretty unusual issue, I would not count on a upgrade fixing it. 
    Can you post one of the text files here? Or send me one of the text files? 
    gmillward@rossvideo.com


    Original Message


  • 11.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago
      |   view attached

    Here one of the text file.

    Rhx for your help



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Attachment(s)

    txt
    29282-INTRADAY.txt   23 KB 1 version
    Original Message


  • 12.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    So, I tested with your file. In the Datalinq Server log window I get:

    ADODB DataLinq Source DataLinq1 Failed opening ADODB dataset.  Check connection string, and check compatability settings in ADODB configuration. (Too many fields defined)

    Apparently, it doesn't like your file - the field limit is probably 255 and your file has over 700. You can create a "Text Datalinq Source" and set "Delimiter" to "#" and it does read it. But this won't help with dynamic file names because you have to enter a file name in the Datalinq setup. You'd have to create a Datalinq for every file (AFAIK).

    I refer you back to my post about manually parsing the file.



    ------------------------------
    JohnCorigliano
    Senior Software Engineer
    ISC
    ------------------------------

    Original Message


  • 13.  RE: datalinq v11 folder with thousand .txt files

    Posted 18 days ago

    This does kinda work for me in the ACE driver. But I have same issue as John with the Jet driver, and he is correct as to why it does not work. 




    Original Message


  • 14.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Strange. I guess a work-around would be to load the files in scripting and parse them manually 🤷‍♂️.

    Dim path As String = <create dynamic path here.....>
Dim lines As String() = System.IO.File.ReadAllLines(path)
For Each line As String In Lines
	Dim columns As String() = line.Split("#")
	<Do something with the data here...>
Next


    ------------------------------
    JohnCorigliano
    Senior Software Engineer
    ISC
    ------------------------------

    Original Message


  • 15.  RE: datalinq v11 folder with thousand .txt files

    Posted 15 days ago

    Hi everyone,
    Thanks! The script is working perfectly now.
    I might try building a gradient brush next, or maybe use a brush as a mask for a gradient material. I'll let you know once the chart area layout is fully built and ready! :)



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 16.  RE: datalinq v11 folder with thousand .txt files

    Posted 20 days ago

    It does sound like a permission issue. 

    You might be able to get the network share mapped to a drive letter if you do this to windows:

    In Registry Editor, locate and then click the following registry subkey:
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System

    Right-click Configuration, click New, and then click DWORD (32-bit) Value.

    Name the new registry entry as EnableLinkedConnections.

    Double-click the EnableLinkedConnections registry entry.

    In the Edit DWORD Value dialog box, type 1 in the Value data field, and then click OK.

    Exit Registry Editor, and then restart the computer.




  • 17.  RE: datalinq v11 folder with thousand .txt files

    Posted 19 days ago

    Thx, if something go wrong, i'll try this.



    ------------------------------
    Benjamin Callu
    Realisateur/Integrateur
    BFMBUSINESS
    ------------------------------

    Original Message


  • 18.  RE: datalinq v11 folder with thousand .txt files

    Posted 20 days ago

    And for future consideration, and for others reading this thread who may find it of interest. 

    In the next release of XPression, version 14.0, later this summer. The Datalinq XML Data source supports this new watch folder mode:

    As you can read in the image, this will allow Datalinq to monitor a folder for XML files (sorry Benjamin not text files) and create tables for each XML file found.