AnsweredAssumed Answered

vrf Masses of data from Excel...

Question asked by antti.suhonen on Jun 10, 2004
Hi,

     Umh, thanks Shawn, just were working on through-file trick...apply
range.copy, create a new wkbk, use parse, save as xlTextMSDOS, open up in
vee, delete --> Works like a lightning ;D

     Just that, my locales are using comma as decimal separator and that
does some funny stuff with e.g. asReal object...How was the thing that was
based on 'obscure' feature of VEE arrays, to replace a portion of (array)
string if it's there...Something to use a sorta transfer array...can't
recall; my head is not as it used to be ;D

BR,
Antti



> -----Original Message-----
> From: Shawn Fessenden [mailto:shawn@testech-ltd.com]
> Sent: 11. keskuuta 2004 12:20
> To: VRF
> Subject: [vrf] RE: Masses of data from Excel...
>
>
> > The size of XLS's are about 50M.
>
> Wow! You're in the realm of "real" data processing. I can
> think of two ways
> to handle this. It depends on the shape of the original data
> and the shape
> of the data you want to process in VEE.
>
> First, if you have to use IDispatch, try to have Excel
> package up all the
> data at once and send it across. Reading item by item will
> only make you
> want to club baby harp seals. I distinctly remember a thread
> about reading
> and writing arrays and I'll attempt to find the relevant
> material. But in
> the mean time...
>
> Consider forgetting about IDispatch. No matter what you do,
> marshaling that
> much data across process boundaries is going to be
> ultra-painful. Save the
> data range to a csv file and use FromFile to read it into VEE.
>
> Well, ok. Three ways. Greg Wale recently mentioned the value
> of executing
> local macros in Excel. Even saving your data range as a csv
> involves at
> least two type transformations: one to text in Excel and one
> back to number
> in VEE. Consider writing a raw data file with an excel macro
> and reading it
> in one fell swoop from VEE with FromFile. This would be one
> of the absolute
> fastest ways to get the job done.
>
> There are ways to eliminate disk operations from the last two
> methods but
> you really have to start considering memory limitations at this point.
> Figure 8 bytes per data point. If the total memory
> requirements of the data
> set exceed roughly 75% of your available physical memory (i.e., total
> physical - that allocated to and used by all running
> processes) then be
> careful about allocating and releasing these huge chunks of memory -
> directly or indirectly. Depending on what's running at the
> time swapping
> could come into play and you're going to get stuck with disk
> operations no
> matter what you do.
>
> I'm going to go look for those data transfer methods I spoke
> of. In the mean
> time, feel free to email me a sample data set and I'll give a
> few things a
> try. Let me know what you want the data to look like in VEE
> as well. Fear
> not! FromFile is more powerful than it looks. There's also
> the option of
> writing a VEE data set directly from Excel and reading that.
> The territory
> is wide open.
> -SHAWN-
>
>
> ---
> You are currently subscribed to vrf as: Antti.Suhonen@elektrobit.com
> To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
> To unsubscribe send a blank email to
> "leave-vrf@it.lists.it.agilent.com".
> To send messages to this mailing list,  email "vrf@agilent.com". 
> If you need help with the mailing list send a message to
> "owner-vrf@it.lists.it.agilent.com".
>



Please note: This e-mail may contain privileged and confidential information
and is intended solely for the addressee. If you are not the intended
recipient or an employee responsible for delivering this message to the
intended recipient, you are hereby notified that any review, publication,
dissemination, distribution, copying or use of the information in this
message is strictly prohibited. If you have received this e-mail by error,
please notify the sender immediately by replying to the message and
subsequently delete it from your system.

E-mail is susceptible to data corruption, interception, unauthorized
amendment, tampering and viruses. We only send and receive e-mail messages
on the basis that we are not liable for any such corruption, interception,
amendment, tampering, viruses, any other reprehensible acts or any
consequences of the aforementioned.


---
You are currently subscribed to vrf as: rsb@soco.agilent.com
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list,  email "vrf@agilent.com". 
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".

Outcomes