Design and implementation of UN Comtrade data sharing platform
-
-
Abstract
United Nations international trade statistics database (UN Comtrade) has provided important data support and application guidance on strengthening the recognition of trade system rules and their driving factors, developing international trade measurement methods, depicting global trade pattern and its changing process. Furthermore, it has been more and more widely applied to global ecological conservation, water-energy-food-land systems, pollution control, energy management, national security and other topics of geographical researches.In this paper, we establish the UN Comtrade data sharing platform with Oracle DBMS based on design of framework and data table structure.This platform is designed to make up for the database’s deficiencies in data sharing methods and retrieval interfaces as well as provide data and tool support for geographic research.We develop an automated data archiving module with Python 3.6 and its Scrapy framework 1.5.2, which achieves a dynamic, stable, and highly fault-tolerant processing of more than 500 million trade records by comprehensively integrating data crawling module, data loading module and nesting multiple error correction methods.In addition, the platform improves the efficiency and scalability of data retrieval instruction execution through range partitioning, partitioned composite indexing, and open ODBC/JDBC interfaces based on the structured characteristics of data.Experiments show that the platform can stably execute different types of retrieval instructions in a concurrent mode of 80 users.By invoking the ODBC/JDBC interface to integrate the calculation process into the retrieval task, the system can more effectively use server-side resources and save time for data transmission, reading and writing with higher efficiency and simplified data processing.Based on the platform, we develop a data query and sharing client and apply it to the retrieval, calculation, grid representation and comparative analysis of the explicit comparative advantage characteristics of Chinese and American products in 2017.The application shows that the platform has the advantages of high efficiency, stable concurrent retrieval efficiency and high scalability.It can provide more convenient, fast, and diverse data sharing services for the calculation and analysis of trade characteristics research.
-
-