English 中文(简体)
合并节点js 和 Python
原标题:Combining node.js and Python

Node.js is a perfect match for our web project, but there are few computational tasks for which we would prefer Python. We also already have a Python code for them. We are highly concerned about speed, what is the most elegant way how to call a Python "worker" from node.js in an asynchronous non-blocking way?

最佳回答

For communication between node.js and Python server, I would use Unix sockets if both processes run on the same server and TCP/IP sockets otherwise. For marshaling protocol I would take JSON or protocol buffer. If threaded Python shows up to be a bottleneck, consider using Twisted Python, which provides the same event driven concurrency as do node.js.

如果您感到冒险,请学习clojure (/a>,>>clojure-py ),您将得到与Java、JavaScript(包括nde.js)、CLR和Python等现有代码运行和操作的相同语言。您只要使用 clojure数据结构就能获得超级连线协议。

问题回答

如果您安排您的 Python 工人进入一个单独的进程( 要么是长期运行的服务器型程序, 要么是需要时产子), 您与它的联系在节点的节点上将是无同步的。 UNIX/ TCP 套接字和stdin/ out/ err 通信在节点中本来就是无同步的 。

使用thoonk.js ,以及 rel=“noreferrer”>thoonk.py 。Thoonk 杠杆 Redis(在模擬钥匙价值商店),为您提供(思想出版物/订阅)、排队和工作模式的交流。

为什么这比 unix 套接字或直接 tcp 套接字更好? 总体性能可能略微降低一点, 但是 Thoonk 提供了一个非常简单的 API, 简化了手动操作套接字。 Thoonk 也使得实施一个分布式计算模型变得微不足道, 该模型允许您扩大 python 工人的比重, 以提高性能, 因为您只是将 Python 工人的新案例翻转出来, 并将其连接到同一个 redis 服务器 。

我还会考虑阿帕奇·特里夫特http://thrift.apache.org/"rel="noreferrer">http://thrift.apache.org/

它可以在几种编程语言之间架起桥梁,效率很高,并且支持进行自动调用或同步调用。见以下全功能:http://thrift.apache.org/docs/faitares/

多语言对于今后的计划可能有用,例如,如果您以后想要在 C++ 中完成部分计算任务,那么使用 Thrift 将其添加到组合中就很容易了。

我建议使用一些工作队列,例如,使用极好的Gearman ,这将为您提供发送背景工作的绝佳方法,一旦他们重新处理后,就会迅速获得结果。

这一点在Digg(许多其他人)大量使用,其优点在于它提供了一种强大、可扩展和有力的方法,使工人能够以任何语言与客户交谈。

<强 > 更新2019年

实现这一点有几种办法,这里是按日益复杂程度排列的清单。

  1. Python Shell, you will write streams to the python console and it will write back to you
  2. Redis Pub Sub, you can have a channel listening in Python while your node js publisher pushes data
  3. Websocket connection where Node acts as the client and Python acts as the server or vice-versa
  4. API connection with Express/Flask/Tornado etc working separately with an API endpoint exposed for the other to query

<强势>Approach 1 Python Shell 简单方法

< 强力> 源代码.js 文件

const ps = require( python-shell )
// very important to add -u option since our python script runs infinitely
var options = {
    pythonPath:  /Users/zup/.local/share/virtualenvs/python_shell_test-TJN5lQez/bin/python ,
    pythonOptions: [ -u ], // get print results in real-time
    // make sure you use an absolute path for scriptPath
    scriptPath: "./subscriber/",
    // args: [ value1 ,  value2 ,  value3 ],
    mode:  json 
};

const shell = new ps.PythonShell("destination.py", options);

function generateArray() {
    const list = []
    for (let i = 0; i < 1000; i++) {
        list.push(Math.random() * 1000)
    }
    return list
}

setInterval(() => {
    shell.send(generateArray())
}, 1000);

shell.on("message", message => {
    console.log(message);
})

< 强度 > 目的地 。 py 文件

import datetime
import sys
import time
import numpy
import talib
import timeit
import json
import logging
logging.basicConfig(format= %(asctime)s : %(levelname)s : %(message)s , level=logging.INFO)

size = 1000
p = 100
o = numpy.random.random(size)
h = numpy.random.random(size)
l = numpy.random.random(size)
c = numpy.random.random(size)
v = numpy.random.random(size)

def get_indicators(values):
    # Return the RSI of the values sent from node.js
    numpy_values = numpy.array(values, dtype=numpy.double) 
    return talib.func.RSI(numpy_values, 14)

for line in sys.stdin:
    l = json.loads(line)
    print(get_indicators(l))
    # Without this step the output may not be immediately available in node
    sys.stdout.flush()

< strong > notes : 制作一个名为用户的文件夹, 该文件夹与源代码. js 文件处于同一级别, 并将目标. py 放在其中 。 不要忘记改变您的虚拟环境 。

我建议您使用“a href=”“https://www.npmjs.com/package/pymport” rel=“no follow noreferrer”>pymport 。

它允许所有Python代码与Node.js完全兼容,包括二元模块:

npm install pymport
npx pympip3 install numpy

然后简单地在 JavaScript 中:

const { pymport, proxify } = require( pymport );
const np = proxify(pymport( numpy ));

const a = np.array([2, 3, 4]);
console.log(a.toString());

(我是作者)

2023年更新

我创建了一个图书馆, 允许您使用带 nodejs 的 python 。

Javascript lbrary:https://github.com/7HR4IZ3/js_bridge

Python 图书馆:https://github.com/7HR4IZ3/py_bridge

示例来自javascript

const { python } = require("js_bridge");
const py = python();

async function myPythonFunction() {
   let math = await py.import("math");
    let result = await math.sqrt(16);
    console.log(result);
}

myPythonFunction();

Python示例

from py_bridge import nodejs

node1 = nodejs(port=7000)
node2 = nodejs(port-7001)

node1.setup(name="my_node")
node2.setup(name="nodejs2") # name parameter is for imports

node1.console.log("Hello from node1")
node2.console.log("Hello from node2")

fs = node1.require("fs") # also supports  from my_node import fs 
print(fs.readSync("./mytext.txt"))




相关问题
Can Django models use MySQL functions?

Is there a way to force Django models to pass a field to a MySQL function every time the model data is read or loaded? To clarify what I mean in SQL, I want the Django model to produce something like ...

An enterprise scheduler for python (like quartz)

I am looking for an enterprise tasks scheduler for python, like quartz is for Java. Requirements: Persistent: if the process restarts or the machine restarts, then all the jobs must stay there and ...

How to remove unique, then duplicate dictionaries in a list?

Given the following list that contains some duplicate and some unique dictionaries, what is the best method to remove unique dictionaries first, then reduce the duplicate dictionaries to single ...

What is suggested seed value to use with random.seed()?

Simple enough question: I m using python random module to generate random integers. I want to know what is the suggested value to use with the random.seed() function? Currently I am letting this ...

How can I make the PyDev editor selectively ignore errors?

I m using PyDev under Eclipse to write some Jython code. I ve got numerous instances where I need to do something like this: import com.work.project.component.client.Interface.ISubInterface as ...

How do I profile `paster serve` s startup time?

Python s paster serve app.ini is taking longer than I would like to be ready for the first request. I know how to profile requests with middleware, but how do I profile the initialization time? I ...

Pragmatically adding give-aways/freebies to an online store

Our business currently has an online store and recently we ve been offering free specials to our customers. Right now, we simply display the special and give the buyer a notice stating we will add the ...

Converting Dictionary to List? [duplicate]

I m trying to convert a Python dictionary into a Python list, in order to perform some calculations. #My dictionary dict = {} dict[ Capital ]="London" dict[ Food ]="Fish&Chips" dict[ 2012 ]="...

热门标签