Find Jobs
Hire Freelancers

Application to Persist Large XML Data Files into an MSSQL Database (Repost)

$30-250 USD

Completed
Posted almost 6 years ago

$30-250 USD

Paid on delivery
I have a number of very large XML documents, and I'd like to have a script made in Java that will read data within the XML document, and persist it into a database with configurable batch sizes. The specific structure of the XML documents may very, but they all have the same basic parent/child structure. For example, we might have: <Parent> <Group_Child> <Child_1>Data<Child_1> <Child_2>Data<Child_2> </Group_Child> <Child_1>Data<Child_1> <Child_2>Data<Child_2> </Group_Child> </Parent> or <Parent> <Group_Child> <Child_3>Data<Child_3> <Child_4>Data<Child_4> </Group_Child> <Child_3>Data<Child_3> <Child_4>Data<Child_4> </Group_Child> </Parent> As you can see above, the first file has two columns, Child_1 and Child_2, while the second file has Child3 and Child4. The application will work as follows in each above case: 1. Parse the XML document to get the list of ALL record element names. Just as with above, we need to review within the entire document to get a cumulative, distinct list of any and all column names; 2. Create a table with the desired structure (e.g. in the above case, tables would be created with the two columns indicated; and 3. Persist all data from the XML documents into the table created. The application should accept three parameters, the first is the filename to be read, the second is batch size, and the third is target table name. For example, if my app name is PersistRMXML, and I want a batch size of 10,000, and the tablename I want to create is SomeTableName, then the command would be: java -jar PersistRMXML [login to view URL] 10000 SomeTableName Upon execution, the application will first read the input file to determine any and all possible column names (each file can contain over one million records, and so the file should not be loaded all into memory at once). STaX can be used to accomplish this. Once this is done, the new table will be created on the target database with the desired structure. The application should contain an [login to view URL] file where the target MSSQL database connection string should be configured. Once the target table is created, the application should read records in batch sizes specified above, and insert read records into memory. Once the desired number of records has been parsed from the XML document, and stored in memory, the application should then persist the data into the new table using the BulkCopy operation of MSSQL. More on Bulk Copy can be found here: [login to view URL] As a final step, the application should validate that the number of records inserted into the target table is the same record count as the number of records persisted. If not, the application should print an error to the log files indicating 'Record Persistence Incomplete - XML record count 11000 - Table record count 50' The application should use a logger, like log4j for logging, and should log all details to the log file. the application should be compiled using the Java 8 platform, and preferable will be distributed as a far jar.
Project ID: 17210801

About the project

9 proposals
Remote project
Active 6 yrs ago

Looking to make some money?

Benefits of bidding on Freelancer

Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
Awarded to:
User Avatar
Hello! Ready to help with your work. I am Java and Android developer with 5 years of work experience. I have a lot of experience with large files. P.S. Thanks for the detailed technical project.
$100 USD in 2 days
5.0 (1 review)
1.6
1.6
9 freelancers are bidding on average $171 USD for this job
User Avatar
Hi sir. I'm very interesting your proposal. I can do it.(xml->mssql). How about working with me? Let me know the result. Regards.
$155 USD in 3 days
5.0 (85 reviews)
7.8
7.8
User Avatar
hi I am really interested in your project I have full experience of java programming for sql db and xml management I could parse your xml file and insert into mssql tables as you want I could satisfy you Everything will be done well if you work with me Thank you
$155 USD in 3 days
5.0 (45 reviews)
6.0
6.0
User Avatar
Hi There, I have read and understand the requirement to insert the XML data in MSSQL using JAVA. I will make it a perfect job. I have 14 years industry experience working on Java platform. Please let me know. Regards, Shatrughna
$350 USD in 5 days
4.9 (34 reviews)
6.1
6.1
User Avatar
I have experience with Java and DB. Java : JPA, EJB, Hibernate, Spring Boot, Spring MVC, internationalization, DI, SOAP, RESTful, XML, JSON. DB: Oracle (admin, PL/SQL, stored procedure, function), MySQL (admin, sql, stored procedure) Linux admin: RedHat CentOS.
$111 USD in 2 days
5.0 (2 reviews)
2.0
2.0

About the client

Flag of UNITED STATES
SCOTTSDALE, United States
5.0
119
Payment method verified
Member since Jul 10, 2006

Client Verification

Thanks! We’ve emailed you a link to claim your free credit.
Something went wrong while sending your email. Please try again.
Registered Users Total Jobs Posted
Freelancer ® is a registered Trademark of Freelancer Technology Pty Limited (ACN 142 189 759)
Copyright © 2024 Freelancer Technology Pty Limited (ACN 142 189 759)
Loading preview
Permission granted for Geolocation.
Your login session has expired and you have been logged out. Please log in again.