Hyperconvergence meets BigData


Formal Metadata

Hyperconvergence meets BigData
Title of Series
Part Number
Number of Parts
Monnerat, Rafael
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
Release Date

Content Metadata

Subject Area
Rafael Monnerat - Hyperconvergence meets BigData This presentation show how to deploy **[Wendelin]**, the free software platform for Big Data & Machine Learning, using **[SlapOS]** , the free software hyperconverged Operating System (hOS). Written in 100% in Python, SlapOS and Wendelin, can create a complete Big Data Infraestruture with out-of-core capabilities ready to use and operate in just few hours. ----- This presentation aims to demonstrate how to use [SlapOS] (Hyperconverged OS) to deploy an entire Big Data Infrastrucure and show how "data life cycle" can be managed with [Wendelin] - covering ingestion, analysis, visualization and weaving it into an application. We'll show how Wendelin and SlapOS could handle acquisition, analysis and exploitation of data, making it a potential solution for IOT scenarios where data is available and needs some logic applied before being presented as web application, possibly on a commercial basis. The agenda of the presentation includes an introduction on SlapOS, as a tool used to deploy a wide range of different services and an introduction of Wendelin, as a tool in order to make out-of-core python applications. After a short introduction, we progress to show the steps to deploy SlapOS infrastructure and later to deploy Wendelin on the just deployed SlapOS, including an use case which shows SlapOS deploying a fluentd instance to ingest data to the Wendelin Database. To conclude, we make a live demo with an Jupiter using out-of-core python to handle wav files stored on Wendelin, and a second short demo on handle computer resources consumption data.
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
Computer animation
although not and my name my name is roughly have someone there I'm going to talk about that I could be the covariance hyper conversion meets Big data I want from from Paris I will bored today
hybrid hydroconversion that we do we say that the west uh and how we deployed Big Data projects using an event how we deployed how we normally low data and then I will start to make a quick demos in the end of the presentation so the goal
of this presentation is so to be a bit more that not necessarily your going to the users that OS not necessarily your going to provide the data with randomly so but then the merger of the 2 that we have been using it it's somehow work here the key that we could imagine for future Ingram beta hybrid competencies we should be data in order to collect out to make the deployments for Big Data Mining and the Internet of Things so how this presentation so this choose to use in this presentation reflects How long necks CDU walks with their customers so makes
it is 1 of the largest open source pollution in Europe despite the fact it's a small company with uh just turn 30 40 employees we could produce the amount of open source software and and I were border 2 of them today this is just what they they stack them going to myself on today and these tools was a war mostly created to uh between real data
and need for a customer it couldn't find an alternative solution for the problem that the test and during the presentation I will give some examples of how uh How was the philosopher who was designed and then there are implemented in the during the evolution of the tool was targeted to cover topics which don't exactly over by other 2 so this is just a list of of i and stock-purchase which is fully open source and then most days and by except if
you went to the 1 which is actually written you will be it's not something that was written by but in the market and we couldn't find and as reliable solution right these days so vendor the course is to provide all just come to the fore by uh
by data which means that we we can process data which is larger than the rest of the computer new is a distributed database for those who knows opiates and distributed pursuant be your fire was it's an open source ERP and not not so exciting and in this presentation slept user is the 2 2 presented follows that resist is
something that we developed to provide a match interconnected mesh networks worldwide for entities to collect data on site this for do Machine learning and another so this was the start of this these latter was it status in 2 thousand 9 or 10 I don't remember what you did today and when he was abused for the 1st time we have we were proposing at that time to put some of those in people's homes so we design a system that could be distributed in a way that could walks in more than 1 data center so we could walks in Amazon
or uh hacks space so the age or any other provider and also to call over there to be able to host uh uh services in the people's home or office and distributed when it starts to work very well and then it's with the Internet of Things and other projects that it's coming up the model and
was going to this level as well as the became a tool that could be standard on machines that we're in encompass trucks to provide mobile-cloud we have a project on going this it can be used a tool tool holster services in other people's boxes like in France there is free boxes so you
could produce equivalent of the week's level as you to work it's been used it buy in there turbines so the wind turbine will collect data in order to inform the when it is there is a need for preventive maintenance and also it's a it's a it's also uses the tool create the Internet of Things
rotors get has verifying installed it and then you can collect data over the several devices that are eventually connected in the natural In order to collect data all managers certain services at your home or
elsewhere so now we become beyond it at the centers can manage at the same time and mobile cloud and uh normalcy and using that as centers using exactly the same system without effective without significant modifications in any of the thoughts and now it's important to to to show that there is that bias so
can provide nodes everywhere and and that it uses a central server for now it's only 1 but in future can be be more than 1 master to control that any amount of computers and devices which we slip
and stuff so just to illustrate a bit so what is select so is that
process come it by whatever the notes in style whatever units available to date as base so there is a tree for components that is based upon score direct that coordinates everything we are basically doubt so we can reconstruct software from from scratch or users some cash automation to uh share already combined software between 2 machines which are operated on the same architecture and we use supervised deal for manage the process on top of it so this is what is present on all machines so in on top of it to you have the subtrees the subtrees is are some kind of think even from group of packages that there but our place the united in a
special way in the system in order to provide binaries to run whatever service that binaries are suppose all and the you can have several configurations in 1 machine people so the same machine can have more than 1 version of my my read your Apache all word processor running at the same time we don't conflicting each other so the softer releases itself don't provide any new year random process is just is just stop at the end of and we have and the softer instances
are and they're the ones that runs the services so you can imagine that when you install a package it only provides the binaries and they only eastern science to tell how their binaries we're run how the
service will be composed ending the suffering instances it's a bit similar has a kind of micro container they're Internet but they are more lights in no way that you don't provide over the head of recall being the same as a group of 5 is everywhere that's why there is this measure of separation so these is represents for example 1 machine running anywhere can be it can be a machine that at the center of all my laptop which I move around anywhere and can be it can be hosted enact ask if there is a need for for a and the 1 1 important things to remark is that it can run uh at the
same time the end old you go machines so a bit like you know what to OpenStack and also other services that we try not to be at viewed machine cannot neutralize it is just process so and the machine can compose many different kind of services in a distributed way so that classic can use more than 1 machine 1 machine has several machines it depends how the comport
with the help of the composition to you configure I will I will later in this presentation I will show in the case of the big data we rendered so based on that is configuration we can provide at the same time
using sharing services sharing computers several so that we can supply several projects and around them all at the same time sharing the machines and they are by this least you can see that there are significantly different in in terms of so we are running for example to the CDN worldwide
which is present in China so we have services in China so we have the TGM lessons for the thank Caroline which is a which is a French project that provides the data for large a French companies at the Institute mean Telecom so we have handling involving data showed
mentioned in the sequence it's being print production to data provide preventive maintenance for wind
turbines in Germany and then we have developmental we have distribution of test nodes
some kind of equivalent of Chongqing distributed in several missions worldwide so we have the automation of we have a system that can produce the EM images so we automate I don't know if people
know or not we have to make to the work that after for generating yams so we can generate previewed and we also use it to provide their chromium West images fall Chromebooks so we have our own and distribution of crime in West which is called by US and we use West also to view the images for ourselves for the persons that 1 and from the main to this wonks its request to uh leveraged away from stellarton everywhere so if you we have 20 different ways to install the same thing based on different architecture as it would be it will require much more afford to the price anything so and come
back to this 1 uh so how we do the deployment of the we have only 1 line in between 1 line installation script that US you if you want to connect to master so if you want to connect to whatever amount so you can tell which 1 you will connecting your device to the machine for example this that thought is connected to the master so I can use most of the great services on my laptop or if you have a mobile cloud it's the same way I can control machines based on master deploying them the point whatever service to whatever machine i it's connected to their so we use a principle which is also a vital to alternate the the set up of the node In the Heat allow us to with the same line the concept of
has very vector like in crumble all or production data center so in this way I danceable taking out there meaning not the particularities of this system that is being started so we can support a very large amount of units distributions for example just by using this come
preferred distribution is not supported for whatever reason we will be happy to act we just add their demand so you don't have to actually connect always to a master you can deploy a stand-alone I stand-alone node by AIDS keeping their questions and running this to Commons here they few types that person figured local you get the computer of figurative tool use that any software that is available in the and this 1 is for the machines politicians and the folder dozens of you don't you you have an API you have to have a winning stance that laborious you also have a command line tool that allows you to supply and request and and use a console to alternate their deployment of the office software that you want to deploy so here is an example of uh request and supplies and applying yourself to release of when told 2 out of computer and then and then I requesting to run 1 instance of this monitoring on on this computer so it's the achievement of a set and when you talk for why are wind turbine for example here
is the vector is here is just variations of the of the
services of the request that you can be done so when you divide his money all you already deployed for entity which is what collects the logs from the machine which leads me to defending so as little as is everywhere and is it's this time that arises in no way that we can put you to anywhere so we we were able to quickly deploy uh diff abandoning stack which is a tool for providing being data and that is is an out of core quieter and dead advantage of users level as in this case it doesn't require hours of set up to me to have a stack so
even now uh and that the scientist's which has no background on Sept opinion clusters can set up and also allows the persons that which are not a data scientist has a different stack also for example psychic learning by out of course distributed database and uh I bite I should the notebook ready-to-use form a some kind of calculation so both sides can that can
benefit from the we set that by not extending time on learning of How to keeping style of mean by on the REM and REM in your clusters by all examples like this so they defending also was designed it to walk on the commodity how so we don't we require a super powerful machine to be deployed so you can and keeping their dimension of
what you're going to do you can make the data with machines that you can bind supermarket for example can buy a few machines I 7 in supermarket that can start to make the data because it's quite easy to find the 1 I 7 with a 16 and 32 get bytes of crime in US and assess these are becoming cheaper and cheaper so you can buy 1 terabyte consistently Tuesday's them quite easily and as deep as everything was designed to be distributed With this little as you can buy several cheap machines and then you have the data you don't have to to expand the 100 thousand euros a buying expensive hardware to start to make have been data so they stock is composed by average harder of
course people get has conditions can combine them by more reliable heartland but the 2 don't we players special service for it so we use less the year P 5 is just as a based tool to probe with narrowed to provide an object database that we are going to manipulate so and psychic learning is to provide a machine learning and other as in order of the features that you can use in the data and the on the at the finest also use it to provide them with 1 already old but to achieve a lent uh
it to the traveling we already had the distributed and actually what's the thing in respective active so we can provide the background in the synchronous approach can do a background in a synchronous programming already by 10 of 12 year so say say started there already doing in a synchronous programming so but this fact is nothing if the data don't right there so you can only do the data if data arrives to their to that tools
so we use to the mostly because it is 1 of the the most reliable tools that exists to today today's we may get best through uh strategy work selecting and there we put down have laptop in a normal our normal of and we let it on that during the weekend that pushing data to uh true vanity of which is the case that we are not analyzing for entity and they could last just 1 registry over a million you need a space of 2 weeks which is very very reluctant reliable because a laptop is turned off In
on all the time because a suspended that's standard in the and a person go home with the laptop in the context from the other network and then they and then it turned off stand in an enabler of ghosts atresia then goes Wi-Fi again so all of these is just lost 1 registry and funded places which can we cannot afford to run our entered the
process we can just run an HTTP and that we can and the weights to the server and can crawl we 1st went to the and what we extend quantity on these cases to the stream of binary data because of the the problems so soon but we can use wave wave
sounds in a way that while the former to the Panamanian plots and that had you out of the so how would we deploy so here we learn how to to request you to whatever
computer money at all to come with friend to and here we in we just the 2 lines can request defended so the full-stack we've all the uh we before all tools and psychic learned work the 2nd learning by in vending pour out of data there is several orders scientific you tools that started on it is available just by typing this to common in whatever no do you want to or if you are in a stand-alone fashion you don't want to connect you want to just to have our unease sensing your at the enemy Amazon you just like this this common and you yet everything so so now we are going to release it was not ready for this conference but soon we're going to release a ready to use images for GM always situ where the should also sound where the call blocks and so on which you can which can provide a ready to use it ready to try would say uh instances of so you don't have
to Spain yourself for the style of the data the so here the 2 the random and the text and social problems and the so here is the
configuration see here is that what I run on year-to-date nodes the data so you generate a 5 which is basically like this we'd say user you get to these folder always science saved a position to know what you already sand or not and then you can take your your data you can have 2 different states for different data as and then send it to work our send to different ingestion policies to classify your shower you can do whatever you want on data and here I just use of the word rendering which is already come with when it got that I just said but if you just just for just French d from Treasury data you you can easily install is justified in a folder then you stay where you are where you were sent into when I want to use the user and which is the best 1 and here you can see that the fonts 6 ways years 6 where and extreme everything to the user to in gesture so in few minutes against African just find in your big data so probably it's also walks through for the order the achievements and some friends to like log patch and
the program to name of other 1 also we tend to be so you can write a Peruvians which are about compatible just by making post and making sure that you you're ready have them you are consistent to when you send the data come back so this is what I
just show and you can be made to develop and so on you buffering memory or if you had that it's too much streaming of Big data can buffer in this and you can just run on 1 like this this quantity let's see and then you have to conservation plan or you can add or or depending on your susceptible for using the Laplace panopticon right a very complex configuration file so takes just few minutes here it's was just what they show when you ran it you just say that you send data and if you are using different proteins fall for example to get his log loss machine consumption that we used to he just use a different you have just to their the part of the source I can show another examples of that tend have
time so how where they they don't roles so just jump green
so the data goes to the user so this is the URI of European provided to be able to do justice target data and by using a fast and put it in created entirely in the box tool be ready to be used that configuration file that you you saw solely to create a portal ingestion that you can use a tool for example if the tag is user depending of the tag you can have right different data streams and not doing anything complex yeah what the right place just putting the same the stream of data I can just go to the extremes as you can just search on so here they have been my worry that they send the idea before the amount of data I can manually a science to overwrite the file that is already for variety of data that is there and I cannot named Fiszman right so you don't have to only rely on the french due to send data you can upload certain data that you have to manipulate all or you can reach
posts for example to put a certain that a certain data stored in that you have then how to use these data so I have this data represents that
several ways sounds that was trained to
to to this computer I just
show the countries that have when they started to do damage so here's a demo so standards that the less everywhere I have my uh my friend please accept opened so if you want if you are a normal speed you can set up everything in 1 day or less and I go to my and more so I hope to have thinks health Roush not make them all but just sold in the thing and believe or not I have I have repeat 6 here even if you don't so uh so instead of fused the normal I quite we just of medium and small extension to their normal i'd 2 notable but today and find tool but uh we just created different turn out that we can use we some magic which can make your life easier and there has to be more reliable when we deal with data so I you assume that you know mainly I and if you get lost to raise their
hand because that would the time to do so some some major and we just this this 1 to fall we just say where
we are going to connect our notebook so it is just the just reference then when you when you finish you get an object called context you can see it as a kind of proxy is not exactly a proxy but whatever you want to contact get did the word in a remote called let's see and have the yes I have thank so I can just call whatever I want I'm doing a remote to that object I don't have to worry so based on because they can get the data that the instrument for the data that you saw it on on the dispatch but of course nobody remembers that ideal for every object that you want to manipulate so you can search the object by using the catalog so you can make queries to the database to know where it is the fact on we can you just used portal catalog then it starts so this is an interesting thought so even if it's out of all you're not calling it the methods on dead like a notebook but you're doing it a remote call to manipulated the objects you can still do bad things so for example these against the Empire tell which is industry and so if there was 1 terabyte data it you get everything as training to young browser not good so the only thing that you have to make to take care of is to use a different uh approach is not not so much different but you have to take care to not load the data all at once when you
want to manipulate here is just 2 examples that we are on call that we
are making the IPython notebook and older data and here we are just managing small chunks of data here is just simple oxygen because they have involved site by 200 the ways that they want then I wanted to make an effort to see we don't know what role the entire data and if you
read the code up this week function you to that is a file as its expected that could use this value the string IU of vitamin however if I use this thing that you I have to load the entire data to give to the string that you that's
why a major disease class as a rock to make gun
out of all right from the multiple stream looks like of 5 the so when I fired pass they're fighting reader to behave like find but we don't loaded interrelated because you can imagine that the data can be 1 terabyte In this way
by using this fine I
can manipulate later one-terabyte defined as it is and average 5 without we don't require you to have 1 terabyte of memory so here I just get 1 channels and I can hear by just saving on running out of so here I'm just saving variance and protein so I'm just starting the reason and making FF FFT T here so I get there I get radiates saving earlier you really get to the array In order to make it a lot of core and then I save it and I've got I can save images to to the database yeah so
here is the lower was their problems finds that invoke adapted to select and the
5 or I can
reject much later when I read it I already process and reported so I can
saved and recovered their that I'm I'm using so the name the 2nd demo which is
now I am going to immolate synchronous processing using likely
OK so the same thing as before long so here I mean it's just like calculation to to see how much later in having this site 30 between the divide and here I just made same cope with as same cooperation saving their calculation 10 by 10 so let I put in background processing time the so I'm just putting data in the background the processing and later time checking for the processing is already finished then I can make the same calculations I again but in doing that kind of market reducing and the using a cluster of consensus instead of the program myself in the level of the IPython notebook so this is the kind of
things that you can do In after 1 day of September so right to do this in this so you could do uh directly so you can follow the tutorial and getting I can make
it available at treated to the site so you can make it at a time and then in the fall of lot data directly in the browser using JavaScript there is a short tutorial and you can use it beeping style and then use the corresponding we don't installed full stop so you can use than any amount of call features uh just in your computer to make a smoker relations which exceeds the around the to
head and your computer I don't think you very much and was extended in the future so now it's I think it's a covered and most the dual the of your coffee the break so it's anyone have questions feel free to to go to possibly and so


  609 ms - page object


AV-Portal 3.9.2 (c7d7a940c57b22d0bc6d7f70d6f13fde2ef2d4b8)