Learning how to work with Big Data comes with a lot a new terminology (and jargon!). In an effort to bring some clarity to what can be a confusing area, the SAGE Campus team created a glossary of Big Data and data science terms. These are what we feel are some of the most important terms and definitions in the field, but it’s by no means a complete list. If we have missed anything that you would like to see included do let us know!
Aggregation – a process of searching, gathering and presenting data
Algorithms – a mathematical formula that can perform certain analyses on data
Analytics – the discovery of insights in data
Anonymization – making data anonymous; removing all data points that could lead to identify a person
API - an application programming interface is a set of subroutine definitions, protocols, and tools for building application software
Application – computer software that enables a computer to perform a certain task
Artificial Intelligence – developing intelligence machines and software that are capable of perceiving the environment and take corresponding action when required and even learn from those actions.
Behavioural Analytics – analytics that informs about the how, why and what instead of just the who and when. It looks at humanized patterns in the da
Big Data - Big data is an all-encompassing term for any collection of data sets so large or complex that it becomes difficult to process them using traditional data-processing applications.
Biometrics - The use of technology to identify people by one or more of their physical traits.
Brontobyte - A brontobyte is a measure of memory or data storage that is equal to 10 to the 27th power of bytes. There are approximately 1,024 yottabytes in a brontobyte. Approximately 1,024 brontobytes make up a geopbyte.
Classification analysis - a systematic process for obtaining important and relevant information about data, also meta data called; data about data.
Cloud computing – a distributed computing system over a network used for storing data off-premises
Clustering analysis – the process of identifying objects that are similar to each other and cluster them in order to understand the differences as well as the similarities within the data.
Comparative analysis – it ensures a step-by-step procedure of comparisons and calculations to detect patterns within very large data sets.
Complex structured data – data that are composed of two or more complex, complicated, and interrelated parts that cannot be easily interpreted by structured query languages and tools.
Computer generated data – data generated by computers such as log files
Concurrency – performing and executing multiple tasks and processes at the same time
Correlation analysis – the analysis of data to determine a relationship between variables and whether that relationship is negative (- 1.00) or positive (+1.00).
Dashboard – a graphical representation of the analyses performed by the algorithms
Data aggregation tools - the process of transforming scattered data from numerous sources into a single new one.
Data analyst – someone analysing, modelling, cleaning or processing data
Database – a digital collection of data stored via a certain technique
Database Management System– collecting, storing and providing access of data
Data centre – a physical location that houses the servers for storing data
Data cleansing – the process of reviewing and revising data in order to delete duplicates, correct errors and provide consistency
Data custodian– someone who is responsible for the technical environment necessary for data storage
Data feed – a stream of data such as a Twitter feed or RSS
Data mining – the process of finding certain patterns or information from data sets
Data modelling – the analysis of data objects using data modelling techniques to create insights from the data
Data set – a collection of data
Data virtualization – a data integration process in order to gain more insights. Usually it involves databases, applications, file systems, websites, big data techniques, etc.)
De-identification – same as anonymization; ensuring a person cannot be identified through the data
Exploratory analysis – finding patterns within data without standard procedures or methods. It is a means of discovering the data and to find the data sets main characteristics.
Exabytes – approximately 1000 petabytes or 1 billion gigabytes.
Extract, Transform and Load (ETL) – a process in a database and data warehousing meaning extracting the data from various sources, transforming it to fit operational needs and loading it into the database
Failover – switching automatically to a different server or node should one fail
Fault-tolerant design – a system designed to continue working even if certain parts fail Feature - a piece of measurable information about something, for example features you might store about a set of people, are age, gender and income.
Graph Databases – they use graph structures (a finite set of ordered pairs or certain entities), with edges, properties and nodes for data storage. It provides index-free adjacency, meaning that every element is directly linked to its neighbour element.
Grid computing – connecting different computer systems from various location, often via the cloud, to reach a common goal
Hadoop – an open-source framework that is built to enable the process and storage of big data across a distributed file system
HBase – an open source, non-relational, distributed database running in conjunction with Hadoop
HDFS – Hadoop Distributed File System; a distributed file system designed to run on commodity hardware
High-Performance-Computing (HPC) – using supercomputers to solve highly complex and advanced computing problems
Histogram - A graphical representation of the distribution of a set of numeric data, usually a vertical bar graph
In-memory – a database management system stores data on the main memory instead of the disk, resulting is very fast processing, storing and loading of the data
Internet of Things – ordinary devices that are connected to the internet at any time anywhere via sensors
Juridical data compliance – relevant when you use cloud solutions and where the data is stored in a different country or continent. Be aware that data stored in a different country has to oblige to the law in that country.
KeyValue Databases – they store data with a primary key, a uniquely identifiable record, which makes easy and fast to look up. The data stored in a KeyValue is normally some kind of primitive of the programming language.
Latency – a measure of time delayed in a system
Load balancing – distributing workload across multiple computers or servers in order to achieve optimal results and utilization of the system
Location data – GPS data describing a geographical location
Log file – a file automatically created by a computer to record events that occur while operational
Machine data – data created by machines via sensors or algorithms
Machine learning – part of artificial intelligence where machines learn from what they are doing and become better over time
Metadata – data about data; gives information about what the data is about.
Multi-Dimensional Databases – a database optimized for data online analytical processing (OLAP) applications and for data warehousing.
MultiValue Databases– they are a type of NoSQL and multidimensional databases that understand 3 dimensional data directly. They are primarily giant strings that are perfect for manipulating HTML and XML strings directly
Natural Language Processing– a field of computer science involved with interactions between computers and human languages
Network analysis– viewing relationships among the nodes in terms of the network or graph theory, meaning analysing connections between nodes in a network and the strength of the ties.
Object Databases – they store data in the form of objects, as used by object-oriented programming. They are different from relational or graph databases and most of them offer a query language that allows object to be found with a declarative programming approach.
Object-based Image Analysis – analysing digital images can be performed with data from individual pixels, whereas object-based image analysis uses data from a selection of related pixels, called objects or image objects.
Operational Databases – they carry out regular operations of an organisation and are generally very important to a business. They generally use online transaction processing that allows them to enter, collect and retrieve specific information about the company.
Optimization analysis - the process of optimization during the design cycle of products done by algorithms. It allows companies to virtually design many different variations of a product and to test that product against pre-set variables.
Pattern Recognition – identifying patterns in data via algorithms to make predictions of new data coming from the same source.
Petabytes - approximately 1000 terabytes or 1 million gigabytes. The CERN Large Hydron Collider generates approximately 1 petabyte per second
Predictive analysis – analysis within big data to help predict how someone will behave in the (near) future. It uses a variety of different data sets such as historical, transactional, or social profile data to identify risks and opportunities.
Privacy – to seclude certain data / information about oneself that is deemed personal
Public data – public information or data sets that were created with public funding
Quantified Self – a movement to use application to track ones every move during the day in order to gain a better understanding about ones behaviour
Query – asking for information to answer a certain question
Re-identification – combining several data sets to find a certain person within anonymized data
Regression analysis – to define the dependency between variables. It assumes a one-way causal effect from one variable to the response of another variable.
RFID – Radio Frequency Identification; a type of sensor using wireless non-contact radio-frequency electromagnetic fields to transfer data
Real-time data – data that is created, processed, stored, analysed and visualized within milliseconds
Scripting - the use of a computer language where your program, or script, can be run directly with no need to first compile it to binary code.Semi-structured data - a form a structured data that does not have a formal structure like structured data. It does however have tags or other markers to enforce hierarchy of records.
Sentiment Analysis – using algorithms to find out how people feel about certain topics
Similarity searches – finding the closest object to a query in a database, where the data object can be of any type of data.
Simulation analysis – a simulation is the imitation of the operation of a real-world process or system. A simulation analysis helps to ensure optimal product performance taking into account many different variables.
Smart grid – refers to using sensors within an energy grid to monitor what is going on in real-time helping to increase efficiency
Spatial analysis – refers to analysing spatial data such geographic data or topological data to identify and understand patterns and regularities within data distributed in geographic space.
SQL – a programming language for retrieving data from a relational database
Structured data – data that is identifiable as it is organized in structure like rows and columns.
Tableau - a commercial data visualization package often used in data science projects. Terabytes – approximately 1000 gigabytes.
Time series analysis - analysing well-defined data obtained through repeated measurements of time. The data has to be well defined and measured at successive points in time spaced at identical time intervals.
Topological Data Analysis – focusing on the shape of complex data and identifying clusters and any statistical significance that is present within that data.
Un-structured data - unstructured data is regarded as data that is in general text heavy, but may also contain dates, numbers and facts.
Variability – it means that the meaning of the data can change (rapidly). In (almost) the same tweets for example a word can have a totally different meaning
Variety – data today comes in many different formats: structured data, semi-structured data, unstructured data and even complex structured data
Velocity – the speed at which the data is created, stored, analysed and visualized
Veracity – ensuring that the data is correct as well as the analyses performed on the data are correct.
Visualization – complex graphs that can include many variables of data while still remaining understandable and readable
Volume – the amount of data, ranging from megabytes to brontobytes
XML Databases – XML Databases allow data to be stored in XML format. The data stored in an XML database can be queried, exported and serialized into any format needed.
Yottabytes – approximately 1000 Zettabytes, or 250 trillion DVD’s. The entire digital universe today is 1 Yottabyte and this will double every 18 months.
Zettabytes – approximately 1000 Exabytes or 1 billion terabytes.