Technology The challenges of storing scientific data

19:25  15 november  2019
19:25  15 november  2019 Source:   techradar.com

Turns out smartphones aren't making millennials grow horns after all

Turns out smartphones aren't making millennials grow horns after all Demonizing smartphones and millennials is a favorite past time among certain segments of the population, and earlier this year there was an excellent opportunity for both when a science journal published a dubious article claiming that using smartphones or tablets led to forward head posture which caused a bony protuberance to grow at the back of the skull. This was quickly picked up and reported as 'smartphones causing young people to grow horns'.

Fundamental to issues around scientific data access/discovery is providing a mechanism to transmit the context of a measurement along with the data . For instance, the regular expression below can be used to definitively identify a string of numbers and hyphens in the CASRN format.

The CODATA Data Science Journal is a peer-reviewed, open access, electronic journal, publishing papers on the management, dissemination, use Defining a "designated user community" for a data collection is essential to good scientific data stewardship. It enables data managers to determine

Karen Ambrose is the database team lead at Francis Crick Institute in St. Pancras in London. We caught up with her at the Percona Live 2019 conference in Amsterdam to understand the complexities involved in managing databases in a scientific installation. Karen has been with the Francis Crick for about five years. She has a background in Bioinformatics and it was during her Masters that she got interested in the application of technology to get better insights into scientific data.

a close up of a computer: database© Provided by Future Publishing Ltd. database

Karen started her career at the Sanger Institute in Cambridge at the time when they were mapping the Human Genome, before moving on to the Francis Crick Institute. The Francis Crick Institute itself came about as a merger between various research organisations including the National Institute of Medical Research (NIMR) and London Research Institute (LRI).

Companies aren't keeping their cloud data secure

  Companies aren't keeping their cloud data secure New research from Thales has shed light on the difficulties of securing sensitive data in the cloud.To compile its 2019 Thales Global Cloud Security Study, the firm surveyed more than 3,000 IT and IT security practitioners from Australia, Brazil, France, Germany, India, Japan, the UK and the US. Of those surveyed, only one in three (31%) organizations believe that protecting data in the cloud is their own responsibility.

A scientific crowdsourcing challenge offers a unique opportunity for scientists to engage with other scientists, and gather insights and new ideas that can Data and results are then collected and verified through a new crowdsourcing method. Initiators of the project recognized that the traditional peer

Computer science (sometimes called computation science or computing science , but not to be confused with computational science or software engineering)

Her first task was to migrate the data from the different databases in the various organisations: “We initially had a time frame I think of about nine months to a year to physically migrate and move into the Francis Crick. And so we have to migrate about 300 databases. But that was in a landscape where the groups weren't entirely moving in one go. So you might have a group, which essentially will talk to a cluster of databases at one site. Half of that group is then moved into the Francis Crick, and the other half is staying in because they have to shut down their lab in order to move. And we've got to make that data available at the new site and the old site.”

  • Databases are migrating to the cloud
  • All things database: Q&A with DataStax's Jonathan Ellis
  • These are the best cloud databases of 2019

What made it even more challenging was that it wasn't just a set of databases assigned to one group that was moving; some of those databases were being shared between five groups that were moving at different times. Karen describes the migration as shuffling chess pieces during which she had to make sure that they don't corrupt any data and that it's available to the teams that were still working on it, with the least amount of downtime, if any.

Scientists suggest creating a detailed 3D map of Earth before we mess everything up even more

  Scientists suggest creating a detailed 3D map of Earth before we mess everything up even more Humans are affecting Earth's climate, and there's a wealth of scientific evidence to support that fact. We've already begun to see some seriously troubling trends that may be associated with our altering of the climate, including mass die-offs of ocean coral, widespread drought, and increasingly powerful and unpredictable storm systems. With all that in mind, scientists from Colorado State University are doing what they can to preserve a record of Earth before things too out of hand. It’s called The Earth Archive, and there’s a good reason why it should be a high priority.

Data Management Challenges of Large-Scale Data Intensive Scientific Workflows -. ewa deelman university of southern. International Scientific Collaborations • Many scientific projects are global collaborations of 100s of partners • Efficient computing and data infrastructures have become critical

The history of string theory spans several decades of intense research including two superstring revolutions. Through the combined efforts of many researchers, string theory has developed into a broad and varied subject with connections to quantum gravity, particle and condensed matter physics

It sounds like a herculean task, and given their strict deadlines would surely have required an army of database wranglers: “There's four of us in the team, including me.”

a circuit board© Provided by Future Publishing Ltd.

Strategising storage

“Over the years we've basically been building a scientific data mountain. Data doesn't get smaller, it just seems to get more complex and large.”

The institute has about 1500 people, including about 1300 scientists and 200 operational staff. There are some 130 lab groups supported by about 18 to 20 Scientific Technology Platforms (STPs) that provide the core services to the lab groups to be able to further their science: “So things like structural biology, and electromagnetic microscopy, high throughput sequencing, scientific computing, of which the database team which I manage is part of. So we provide a core service to the rest of the Institute.”

The Department of Energy Has Way Too Much Data for Regular Old Computers

  The Department of Energy Has Way Too Much Data for Regular Old Computers So it needs some money to make its machines super.The DOE is expected to ask Congress for between $3 billion and $4 billion over the next decade to fund a new type of supercomputer that it's referring to as a next-generation "exascale" machine. It's all to better handle the insane amount of data it has collected and stored on hard drives and basically turn its fleet of computers into AI machines.

Learn about the scientific method, with these explanations of each step of the process, the variables involved, and why these steps are important. The scientific method is a systematic way of learning about the world around us and answering questions.

A data scientist worth his salt uses applications that help him surmount the three key challenges to his job. It is becoming increasingly apparent that data scientists need to demonstrate skills necessary to convert data -based scientific inference into accessible, actionable insights for business and upper

“For us, it's very much about the data that comes off these instruments”, Karen tells us. Besides making sure they provide the right platform to help scientists investigate the raw data that comes off the machines, a major task for Karen and her team is to store the data efficiently: “We need to work out what can we contain within the storage that we have within the institute, and also what other strategies do we need to incorporate, in terms of maybe looking at cloud, to help us provide the scientific insights that a particular lab group requires.”

The first challenge, she tells us, is to manage and secure all the generated data: “If people generate data, they generally want to keep everything, because you never quite know when you might need it. But we can't physically keep everything.” So her team works with the lab groups to identify the important data and separate it from the data that can be generated.

The next challenge is performance. While for some scientists throughput isn't important as long as they can access the data, for others performance is important: “We're always looking how can we best design their database, how does their data need to be structured so that it will be performant.” Once again, the solution Karen says comes up in discussions  with the labs to understand what they need to achieve from the data.

Nigerian entrepreneur Temie Giwa-Tubosun wins Jack Ma's African business hero award

  Nigerian entrepreneur Temie Giwa-Tubosun wins Jack Ma's African business hero award A Nigerian entrepreneur has taken home the top prize at the Jack Ma Foundation's first annual $1 million prize for African businesses. © Lifebank Temie Giwa-Tubosun after the completion of the proof of concept project in Ethiopia Temie Giwa-Tubosun walked away with the top $250,000 cash prize from the $1 million available from the Africa Netpreneur Prize Initiative (ANPI), started by Chinese investor Jack Ma. The organization says it will award a $1m grant to 10 African entrepreneurs every year for the next 10 years.

a close up of a black keyboard© Provided by Future Publishing Ltd.

The Open Source advantage

The Francis Crick Institute uses various types of databases. While for the enterprise side of things, they use Oracle or SQL Server, Karen tends to steer the science groups towards open source databases. The Institute uses relational databases like MySQL and Postgres, but is starting to explore NoSQL databases like MongoDB, Neo4j, Cassandra, and others. She’s particularly keen on investigating Neo4j because “it's interesting in terms of how it graphs the relationships between data.”

Karen also likes working with open source databases because of their open developmental model: “If you come up with something, a new problem that you want to solve, it's a lot easier to be able to talk to all the community to be able to come up with a solution. They're always innovating, always pushing things forward. So you never feel like you're always going to be confined by stagnant release process.”

  • We've also highlighted the best database software of 2019

An end to slip-ups with Google Assistant .
If you make a mistake, it is now possible to cancel Google intelligent assistant orders using a simple voice command. The newly updated Google Assistant help page has revealed that it is now possible to cancel your most recent voice command by simply saying "Hey Google, that wasn't for you." Using this feature, you can prevent Google Assistant from storing commands or pieces of information that were given by accident. Let's not forget that itThe newly updated Google Assistant help page has revealed that it is now possible to cancel your most recent voice command by simply saying "Hey Google, that wasn't for you.

—   Share news in the SOC. Networks

Topical videos:

usr: 4
This is interesting!