如何自动化API获取数据请求?当使用web sockets

As far as I know Web Sockets allows bi-directional communication. and web sockets (for example: Socket.io) connections are always open. so, whenever new data has arrived data should be automatically pushed to the view via socket.

据我所知，Web Sockets允许双向通信。并且web套接字(例如:Socket.io)连接始终是打开的。因此，每当新数据到达时，数据都应该通过套接字自动推送到视图。

but in below code I am using set_interval to make a http.get call. and set_interval is called once every 1 second.

但是在下面的代码中，我使用set_interval创建一个http。得到调用。set_interval每1秒调用一次。

now, doing these does not give a real-time feel that is, the new data is pulled once every 1 second. which is statically defined.

现在，这样做并没有提供实时的感觉，新的数据每1秒被拉一次。这是静态定义的。

in-short, I want to automate what set_interval does in below code. I don't want a static fetch interval value. This is because at-times stock price could change within 100ms and at times it would change once in few seconds.

简而言之，我想让set_interval在代码下面实现自动化。我不想要一个静态的取回间隔值。这是因为有时股价会在100毫秒内发生变化，有时几秒钟就会发生变化。

Now, if I set interval to 1 sec, that is make a call every 1 second. the real feel of high fluctuation in market move would not be seen.

现在，如果我将interval设置为1秒，那就是每1秒打一次电话。市场波动剧烈的真实感觉是看不到的。

I am not sure how usually developers fetch data in IOT applications. for example car is monitored in real-time and let's say speed of the car is fetched in real time and graphed on a web or mobile application.

我不确定开发人员如何在物联网应用程序中获取数据。例如，汽车被实时监控，假设汽车的速度被实时获取并在web或移动应用程序上显示出来。

How do I achieve something similar like that in Stock Ticker? I want to simply plugin the application to an API and when new data arrives instantly push it to all the viewers (subscribers) in real-time.

如何在股票行情中实现类似的目标?我想简单地将应用程序插入到一个API中，当新的数据到达时，即时将它推送给所有的观众(订阅者)。

Code below

////
// CONFIGURATION SETTINGS
////
var FETCH_INTERVAL = 1000;
var PRETTY_PRINT_JSON = true;

////
// START
////
var express = require('express');
var http = require('http');
var https = require('https');
var io = require('socket.io');
var cors = require('cors');

function getQuote(socket, ticker) {
    https.get({
        port: 443,
        method: 'GET',
        hostname: 'www.google.com',
        path: '/finance/info?client=ig&q=' + ticker,
        timeout: 1000
    }, function(response) {
        response.setEncoding('utf8');
        var data = '';

        response.on('data', function(chunk) {
            data += chunk;
        });

        response.on('end', function() {
            if(data.length > 0) {
                var dataObj;

                try {
                    dataObj = JSON.parse(data.substring(3));
                } catch(e) {
                    return false;
                }

                socket.emit(ticker, dataObj[0].l_cur);
            }
        });
    });
}

I am making a call to method getQuote depending on FETCH_INTERVAL set above

我根据上面的FETCH_INTERVAL集调用getQuote方法

function trackTicker(socket, ticker) {
    // run the first time immediately
    getQuote(socket, ticker);

    // every N seconds
    var timer = setInterval(function() {
        getQuote(socket, ticker);
    }, FETCH_INTERVAL);

    socket.on('disconnect', function () {
        clearInterval(timer);
    });
}

var app = express();
app.use(cors());
var server = http.createServer(app);

var io = io.listen(server);
io.set('origins', '*:*');

app.get('/', function(req, res) {
    res.sendfile(__dirname + '/index.html');
});

io.sockets.on('connection', function(socket) {
    socket.on('ticker', function(ticker) {
        trackTicker(socket, ticker);
    });
});

server.listen(process.env.PORT || 4000);

Edits - Update

Okay, so I would need real-time feed. (this bit is sorted)

好的，我需要实时提要。(这一点是排序)

As far as I know, Real-time feeds are quite expensive and buying 10,000+ end points for each online client is quite expensive.

据我所知，实时的feed是非常昂贵的，并且为每个在线客户购买10,000+的端点是相当昂贵的。

1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.

1)如何利用实时feed服务于1000万终端用户?我是否可以使用web sockets, Redis，发布/订阅，广播或一些技术，将实时feed复制到大量用户?我想要一个高效的解决方案，因为我希望尽可能降低实时数据传输的成本。

How do I tackle that issue?

我该如何处理这个问题?

2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..

2)是的，我知道轮询需要在服务器端进行，而不是在客户端(为了避免为每个客户端执行轮询)。那么我需要使用什么技术呢?尚,复述,pub / sub等。

I have API URL and a token to access the API.

我有API URL和一个访问API的令牌。

3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.

3)我不仅需要获取数据并将其推送给终端用户。但是我需要对获取的数据进行一些计算，也需要从Redis或数据库中提取数据并对其进行计算，然后将其推到视图中。

for example:

例如:

1) data I get in real-time market feed {"a":10, "b":20}
2) get data from DB or Redis  {"x":2, "y":4} 
3) do computation : z = a * x + b * y
4) finally push value of z in the view. 

How do I do all these in real-time at the same-time push it to multiple clients? Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.

如何在同一时间内实时完成所有这些工作，将其推送到多个客户端?你能和我分享一下路线图吗?我得到了第一个获得实时数据的难题。

1 个解决方案

#1

1) How do I make use of real-time feed to serve 1000s of end users? Can I use web sockets, Redis, publish/subscribe, broadcasting or some technology that copies real-time feed to tonnes of users? I want a efficient solution because I want to keep the expense of real-time data feed as low as possible.

1)如何利用实时feed服务于1000万终端用户?我是否可以使用web sockets, Redis，发布/订阅，广播或一些技术，将实时feed复制到大量用户?我想要一个高效的解决方案，因为我希望尽可能降低实时数据传输的成本。

How do I tackle that issue?

我该如何处理这个问题?

To "push" data to browser clients, you would want to use a webSocket or socket.io (built on top of webSockets). Then, anytime your server knows there's an update, it can immediately send that update to any currently connected client that is interested in that info. The basic idea is that the client connects to your server as soon as the web page is loaded and keeps that connection open for as long as the web page(s) are open.

要将数据“推送”到浏览器客户端，您需要使用webSocket或socket。io(构建在webSockets之上)。然后，只要服务器知道有更新，它就可以立即将更新发送到任何对该信息感兴趣的当前连接的客户端。基本的想法是，当web页面加载后，客户机就会连接到您的服务器，并且只要web页面是打开的，就可以保持该连接打开。

2) Yes, I understand polling needs to be done on server side and not on a client-side (to avoid doing polling for each client). but then what tech do I need to use? websockets, redis, pub/sub etc..

2)是的，我知道轮询需要在服务器端进行，而不是在客户端(为了避免为每个客户端执行轮询)。那么我需要使用什么技术呢?尚,复述,pub / sub等。

It isn't clear to me what exactly you're asking about here. You will get updated prices using whatever the most efficient technology is that is offered by your provider. If all they provide is http calls, then you have to poll regularly using http requests. If they provide a webSocket interface to get updates, then that would be preferable.

我不清楚你到底在问什么。您将得到最新的价格，使用任何最有效的技术是由您的供应商提供。如果它们提供的都是http调用，则必须定期使用http请求进行轮询。如果他们提供了一个webSocket接口来获取更新，那就更好了。

There are lots of choices for how to keep track of which clients are interested in which pieces of information and how to distribute the updates. For a single server, you could easily build your own with just a Map of stock prices where the stock symbol is the key and an array of client identifiers is the value in the Map. Then, any time you get an update for a given stock, you just fetch the list of client IDs that are interested in that stock and send the update to them (over their webSocket/socket.io connection).

关于如何跟踪哪些客户对哪些信息感兴趣以及如何发布更新，有很多选择。对于单个服务器，您可以轻松地构建自己的股票价格图，其中股票符号为键，客户端标识符数组为映射中的值。然后，每当您获得给定股票的更新时，您只需获取对该股票感兴趣的客户端id列表，并通过它们的webSocket/socket将更新发送给它们。io连接)。

This is also a natural pub/sub type of application so anyone of the backends that support pub/sub would work just fine too. You could even use an EventEmitter where you .emit(stock, price) and each separate connection adds a listener for the stock symbols they are interested in.

这也是一种自然的pub/sub类型的应用程序，因此任何支持pub/sub的后台都可以正常工作。您甚至可以使用EventEmitter (stock, price)和每个单独的连接都为它们感兴趣的stock符号添加一个侦听器。

For multiple servers at scale, you'd probably want to use some external process that manages the pub/sub process. Redis is a candidate for that.

对于大规模的多个服务器，您可能需要使用一些外部进程来管理发布/子进程。Redis就是其中的一个候选者。

3) I am not just in need to fetch the data and push it to end users. But I would need to do some computation on the fetched data, will need to pull data from Redis or database as well and do calculations on it then push it to the view.

3)我不仅需要获取数据并将其推送给终端用户。但是我需要对获取的数据进行一些计算，也需要从Redis或数据库中提取数据并对其进行计算，然后将其推到视图中。

I don't really see what question there is here. Pick your favorite database to store the info you need to fetch so you can get it upon demand.

我不知道这里有什么问题。选择您最喜欢的数据库来存储您需要获取的信息，以便您可以根据需要获取它。

How do I do all these in real-time at the same-time push it to multiple clients? Can you share a roadmap with me? I got the first piece of the puzzle getting real-time datafeed.

如何在同一时间内实时完成所有这些工作，将其推送到多个客户端?你能和我分享一下路线图吗?我得到了第一个获得实时数据的难题。

1. Real-time data feed.
2. 实时数据提要。
3. Database to store your meta data used for calculations.
4. 数据库存储用于计算的元数据。
5. Some pub/sub system, either home built or from a pre-built package.
6. 一些酒吧/分系统，要么是自制的，要么是预制的。

Then, follow this sequence of events.

然后，遵循这个事件序列。

1. Client signs in, connects a webSocket or socket.io connection.
2. 客户端登录，连接一个webSocket或socket。输入输出连接。
3. Server accepts client connection and assigns a clientID and keeps track of the connection in some sort of Map between clientID and webSocket/socket.io connection. FYI, socket.io does this automatically for you.
4. 服务器接受客户端连接并分配clientID，并在clientID和webSocket/socket之间的某种映射中跟踪连接。输入输出连接。通知你,插座。io自动为你做这个。
5. Client tells server which items it wants to monitor (probably message sent over webSocket/socket.io connection.
6. 客户端告诉服务器要监视哪些项(可能是通过webSocket/socket发送的消息)。输入输出连接。
7. Server registers that interest in pub/sub system (essentially subscribing the client to each item it wants to monitor.
8. 服务器注册对pub/sub系统感兴趣的内容(本质上是将客户端订阅到它想要监视的每个项目。
9. Other clients do the same thing.
10. 其他客户也做同样的事情。
11. Each time client requests data on a specific item, the server makes sure that it is getting updates for that item (however the server gets its updates).
12. 每次客户端请求特定项上的数据时，服务器都会确保它正在获取该项的更新(不过服务器会获得更新)。
13. Server gets new info for some item that one or more clients is interested in.
14. 服务器获取一个或多个客户感兴趣的项目的新信息。
15. New data is sent to pub/sub system and pub/sub system broadcasts that information to those clients that were interested in info on that particular item. The details of how that works depend upon what pub/sub system you choose and how it notifies subscribers of a change, but eventually a message is sent over webSocket/socket.io for the item that has changed.
16. 新的数据被发送到pub/sub系统，pub/sub系统将该信息广播给那些对该特定项目的信息感兴趣的客户。其工作方式的细节取决于您选择的发布/子系统以及它如何通知订阅者更改，但最终消息是通过webSocket/socket发送的。io对于已更改的项。
17. When a client disconnects, their pub/sub subscriptions are "unsubscribed".
18. 当客户端断开连接时，他们的pub/sub订阅是“未订阅”的。