Transcripción
Extracto de la transcripción automática del vídeo realizada por YouTube.
good afternoon everybody we're going to get started with the next talk I'd like to deduce Kevin who's going to be talking about building a high-throughput systems with Python all right so yeah again my name is Kevin I worked for a company called
tell apart we build a data platform for retailers and we deal with some fairly high high traffic rates low latencies and we use Python to do pretty much all of it so today I'm going to talk about a project called taba which is a distributed event aggregation
service basically what it does is it helps you instrument your code by putting calls to record events and then those get aggregated into a central location where you can then query them so for example we can record the the particular wind price of a type of
bid and then query it for for example the 10 minute sliding window and some percentiles so top is built using Python it uses Redis as a back-end store we have a cluster of those instances it makes heavy use of G event and does some optimizations using scythes
on and it does a fairly high ingest rate uses multiple over a hundred different processors so it's a relatively large scale deployment so instead of talking today about the the architecture and how its implemented I thought it'd be more useful to talk
about some of the lessons learned when trying to build large-scale distributed systems using Python so lesson number one is get your data model right so any given problem has a number of different ways you can solve it number of different ways that you can
model it and the one you go with really limits or informs what your application is capable of doing and some very subtle differences in the way that you model a program can have massive impacts on your ability to perform later and once you've committed
to a model it is extremely important it's extremely difficult to go back and change it especially in a Jewess tributed system where you have things deployed all over the place and you have protocols between different components it's very hard to change
so at a high level this is kind of what the data model for tada looks like you have a large stream of events which are generated by clients those get aggregated into these state objects one per tag which is essentially the metric and the source and those get
stored in the database and then when you perform a query the states that match that query get combined into an aggregate those different objects look like this so an event is basically a tuple of the the name the type the time and some payload and this in
this example of a moving interval counter the state is basically some bucket eyes counts and totals over to certain time range and those states are stored as compact efficient binary objects so that because there they go over the wire a lot they get edited
very often so they're extremely efficient in terms of space and mutability the aggregates are responses to queries so they're the friendly json text that you want to consume by another service so one way we can go we can respond to queries go between
states and aggregates is to convert each state into an aggregate to the simpler representation and then combine those together so require that every counter type or every metric type has a way to combine those aggregates and this seems like a reasonable thing
to do these aggregate objects are much simpler to their much simpler schema so there be easier to implement a way to combine them however there's a problem with this approach and if you're sharp you may have picked up on it that's you can't
[ ... ]
Nota: se han omitido las otras 1.688 palabras de la transcripción completa para cumplir con las normas de «uso razonable» de YouTube.