THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Goals
Enhanced auto-complete for Python 2, 3. This enhanced auto-complete must be able to:
- complete functions and classes that are imported from a module: eg from datetime import dat
- complete module names: eg from dateti
- complete functions, variables and classes available in the current scope (either defined in the scope, from the standard library or imported)
- complete methods and properties of a class instance
- complete methods and properties of a class within it (ie self.)
- complete functions and classes from an imported module eg re.matc
This file can be used to test the behavior of this feature.
The completion features should rely on Jedi. Jedi is a python library that does static analysis of Python code to provide the relevant code completions. It is used by other project to provide code completion like the Python plugin for Emacs or VSCode. Using Jedi has many advantages for us:
- No need to develop completion ourselves
- Good community support
- Project already known and used
- Support Python 2 and Python 3
The main downside is that we need to use a Python process to do the completion.
The development of this feature is currently happening on bitbucket under a jedi bookmark. This is an ongoing process. The feature can be tested on a 8.1 release of Netbeans with the nbms available here. The list of issues with the current nbms is available here.
In addition to the completion feature, the nbms provode code validation with pyflakes and linting with pep8. The reason behind this is that they work the same way (in a Python process) and will allow us to improve the support for python 3 (the current lint and error checking are not compatible with Python 3. For instance, print 3 is reported as valid python 3 code and print(3, file=stderr) is reported as invalid).
Background and strategic fit
- This is a competitive feature
- Very useful while programming and a must have for big projects
- Improve support for python 2 and python 3
- Less pains for the user: no need to use another IDE for python
Assumptions
Requirements
| # | Title | User Story | Importance | Notes |
|---|---|---|---|---|
| Basic support | Meet all requirement of enhanced completion defined earlier. No stackstrace, no apparent bugs. | Must Have | Done | |
| Activation | Jedi completion must be turned on by default within the languages option like for Java (Options > Editor > Code Completion then on the select list Python). | Must Have |
Done | |
| On the fly | Currently the user must use CTRL+SPC to activate completion. This must be available while typing like for Java or Javascript if the user choose to. This feature must be turned off by default like in Java and JavaScript. | Must Have | Done | |
| Options for jedi | Jedi supports a wide range of options. The user should be able to tune them. | Good to have | Very likely to be pushed to a future release. | |
| Options for pep8 | The user should be able to tune the options of Pep8 globally, per project. The settings should be detected automatically if a project contains the "proper" configuration file. This will allow the user to use different rules than the default ones if he/she wishes. | Good to have. | Future release | |
| Options for pyfalkes | The user should be able to tune the options of pyflakes globally, per project. The settings should be detected automatically if a project contains the "proper" configuration file. This will allow the user to use different rules than the default ones if he/she wishes. | Good to have. | Future release | |
| Improve navigator support for Python 3 files | On python 3 files with yield from navigator is empty. This should be improved | Good to Have | Future release | |
| Use Jedi to complete function signatures | This would allow the user to know what parameters are expected for a function. We should also do like in Java: set the focus on the first parameter, allow the user to enter it, once it is correct, the user presses the RETURN key and we switch to the edition of the first paramter. | Good to Have | Future release See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.call_signatures | |
| Use Jedi to provide navigation in source files | The user CTRL+Click on a function/class/variable name and we take the user to the definition. | Good to Have | Future release See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.goto_definitions s = jedi.api.Script(source='def t():\n passprint(t())', line=3, column=6, path='/tm/test.py') r = s.goto_definitions()[0] r.line, r.column | |
| Use Jedi for refactoring | The user select a function/class/variable and rename it with a refactoring tool based or Script.usages to rename all the occurrences correctly. | Good to Have | Future release See: https://jedi.readthedocs.io/en/latest/docs/plugin-api.html#jedi.api.Script.usages |
User interaction and design
- Syntax checking and linting are done automatically when the editor is idle for a while.
- Jedi completion is activated by default
- Code completion works on CTRL+SPC by default. The user can use an option to enable it on the fly.
Add design for option panels.
Questions
Below is a list of questions to be addressed as a result of this requirements document:
| Question | Outcome |
|---|---|
| How do we make this compatible with Jython? | One solution would be to ask the jedi developer to add support for Jython but the best way would be to report the missing package pydoc_data to the Jython project so they can implement it. That would benefit them since it will improve their compatibility with Python code. There's currently no mention of it in their bugtracker: http://bugs.jython.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status&status=-1%2C1%2C3&%40search_text=pydoc_data&submit=search+in+open+issues. |
| or, how will CC work when Jython is used? | Currently, the "standard" although incomplete completion still works. That's the only completion mechanism we have now. Maybe it can be improved. |
| How will Jedi be packaged? | Currently Jedi is package in contrib/python.jedi, pyflakes in contrib/python.pyflakes and pep8 in python.pep8. The relevant python files are installed because of a patch in main. This is similar to how Jython is packaged. |
| Will packaging be dependent on Python version? | Jedi, pyflakes and pep8 are design to run on any supported version of python (2.7 and >= 3.3). So we can package only one version of each software. If something doesn't work on a version of python, we should report the bug upstream and update our package. |
Support on Jython
Jython is a python interpreter written in Java. Since it is not yet fully compatible with Python, some programs may not run. In this section, we detail how the auto-complete feature is impacted by this.
| Features | Supported? | Notes |
|---|---|---|
| PEP8 | YES | |
| Pyflakes | POSSIBLE | Would require more testing. I (Jenselme) tested a month ago and I got an error but today (2016-04-18) I did a test with `./jython-2.7.0/bin/jython python_parse_process.py <<< '{"path": "python_parse_process.py", "source": "import sys\nprint 789\nprint zut\nprint(789, file=sys.stdout)\n"}'` an it seemed to work. However, if we use pyflakes classes, functions names and function parameters are not correctly highlighted. To do this highlighting, we rely on Jedi which doesn't work on Jython. I think that for now, we better rely on "standard" parsing with Jython. |
| Jedi | NO | Jython doesn't implement pydoc_topics (python 2.7 and above) which is required for Jedi to work. See this feature request to fellow the implementation of this feature by jython. |
Not Doing
- List here what we are not doing with the reasons…
Notes on the implementation
The completion services were inspired by their equivalent in nbts a NetBeans plugin for the TypeScript language by Everlaw. Many thanks to them for their work and inspiration.
The completion/syntax analysis features are provided by Python daemons running in the background. We have the following daemons:
- python_lint_process provided by ./python.pep8/external/python_lint_process.py and relying on pep8. It provides style validation according to the PEP8 style rules. It is used by python.source/src/org/netbeans/modules/python/source/PythonParseProcess.java in python.source/src/org/netbeans/modules/python/source/PythonParser.java. The pep8.py script is currently tracked by mercurial.
- python_parse_process provided by ./python.pyflakes/external/python_parse_process.py and relying on pyflakes. It provides syntax validation for Python2 or Python3. It is used by python.source/src/org/netbeans/modules/python/source/PythonParseProcess.java in python.source/src/org/netbeans/modules/python/source/PythonParser.java. Pyflakes will be downloaded during the build process.
- python_completion_process provided by ./python.jedi/external/python_completion_process.py and relying on jedi. It provides completions. It is used by python.editor/src/org/netbeans/modules/python/editor/completion/PythonCompletionProcess.java in python.editor/src/org/netbeans/modules/python/editor/PythonCodeCompleter.java. Jedi will be downloaded during the build process.
- python_analyse_process provided by ./python.jedi/external/python_analyse_process.py and relying on jedi. It provides parsing capabilities for Python 2 and Python 3. It is used by python.source/src/org/netbeans/modules/python/source/PythonAnalyseProcess.java in python.editor/src/org/netbeans/modules/python/editor/PythonCodeCompleter.javapython.source/src/org/netbeans/modules/python/source/PythonParser.java to provide code highlighting when we cannot rely on Jython. Jedi will be downloaded during the build process.
This services are started if they are enabled in the options when opening a Python project or file. We launch one service per Python interpreter (the services are reused between projects/files with the same interpreter). We use the interpreter registered in the project properties to launch the services. Once all projects or files associated with an interpreter are closed, the services associated with this interpreter are stopped. The association between services, interpreters and projects/files is done in a static map in python.core/src/org/netbeans/modules/python/api/PythonProcess.java. Projects and files are registered with a call to PythonProcess.registerProject(projectDirectory.getPath(), platform.getInterpreterCommand()); and unregistered with a call to PythonProcess.unregisterProject(projectDirectory.getPath());. When the user changes the interpreter in the options, the mapping is updated and the proper services are stopped/started.
The Python scripts are designed not to crash (we catch all exceptions). They must always provide an answer (an empty one in case of a problem in the scripts) in order to prevent the IDE to hang.
Credits
This page was imported from the nbpython wiki (https://nbpython.atlassian.net/wiki/spaces/N8/pages/10321925/Enhanced+autocomplete+for+Python+2+3+and+imports). Contributors:
- Lou Dasaro
- Julien Enselme