New preprocessor #2729

ffissore · 2015-03-04T11:20:12Z

This PR introduces a new preprocessor, based on two binary tools: coan and (a modified version of) ctags

coan is use to resolve #define #ifdef et al, producing a clean sketch.
If your sketch has

#define DEBUG 0
#if DEBUG
#include "debug.h"
myDebugFunction("hello debug");
#endif

the resulting sketch will be

#define DEBUG 0

ctags is used to collect function definitions and to generate their prototypes. If a prototype is already defined, it will not be generated.
If your sketch has

void firstFunction() {
  // code here
}
void secondFunction();
void secondFunction() {
  //code here
}

the resulting sketch will be

void firstFunction();
void firstFunction() {
  // code here
}
void secondFunction();
void secondFunction() {
  //code here
}

coan and ctags are defined in a parent platform.txt (currently located in IDE_FOLDER/hardware)

A new mechanism for loading platform.txt allows to group key value pairs common to multiple platforms or packages into separate files. Those pairs will be merged into children platform.txt. The key value pair nearest to boards.txt wins and overwrites pairs with the same key.
An example is IDE_FOLDER/hardware/arduino/sam/platform.txt, where coan command line is customized with sam specific parameters.
This keep users that have defined a 3rd party platform based on the avr one from having to update it adding the new tools definitions, and lowers the burden of those that started from the sam one.

This PR fixes lots of issues: see Component: Preprocessor The Arduino sketch preprocessor converts .ino files into C++ code before compilation

Closes #2636

Wackerbarth · 2015-03-09T13:40:57Z

There is a problem with the function prototype generation.

byte program (const byte b1, const byte b2 = 0, const byte b3 = 0, const byte b4 = 0)
  {
   /* ... */
  return b4;
  } // end of program

generates the prototype

byte program(const byte b1, const byte b2 = 0, const byte b3 = 0, const byte b4 = 0);

however that causes the compiler to fail

Atmega_Board_Detector.ino: In function 'byte program(byte, byte, byte, byte)':
Atmega_Board_Detector.ino:275:85: error: default argument given for parameter 2 of 'byte program(byte, byte, byte, byte)' [-fpermissive]
Atmega_Board_Detector.ino:265:6: error: after previous specification in 'byte program(byte, byte, byte, byte)' [-fpermissive]

ffissore · 2015-03-09T14:22:24Z

What is the correct prototype?

mastrolinux · 2015-03-09T14:26:41Z

byte program(byte, byte, byte, byte)

Is the correct prototype.

mastrolinux · 2015-03-09T14:27:22Z

or with "const" if you like but without the default arguments

ffissore · 2015-03-10T09:20:49Z

Please ignore previous (deleted) build message. A new build with a fix is backing

ffissore · 2015-03-10T10:21:02Z

Ok I've fixed the issue. Given function

byte program (const byte b1, const byte b2 = 0, const byte b3 = 0, const byte b4 = 0)
  {
   /* ... */
  return b4;
  } // end of program

the correct preprocessor result should have the prototype specify the default values and the implemention just mention the var names, like

byte program (const byte b1, const byte b2 = 0, const byte b3 = 0, const byte b4 = 0);

byte program (const byte b1, const byte b2, const byte b3, const byte b4)
  {
   /* ... */
  return b4;
  } // end of program

However this would mean modify your code and we will NOT do that.

Instead, we now generate this code

byte program (const byte b1, const byte b2, const byte b3, const byte b4)

byte program (const byte b1, const byte b2 = 0, const byte b3 = 0, const byte b4 = 0)
  {
   /* ... */
  return b4;
  } // end of program

Where default values are in the implementation, not in the prototype. This is not correct, but compiles and works fine.

If you want to have the correct version, you need to type the prototype yourself: this new preprocessor will not generate prototypes when they are already specified.

ArduinoBot · 2015-03-10T15:41:55Z

Build failed.

ffissore · 2015-03-13T12:24:05Z

I've just found out it doesn't work if you have a configuration #define into a file included by the sketch and your sketch checks it with #ifdef. Coan won't read the included file and hence the #ifdef will result to false. I'm going to fix this in the upcoming days

supuflounder · 2015-03-13T19:33:01Z

OK, I downloaded the PR-2729-BUILD-219-macosx.zip, which contained only the Arduino executable. I moved this to the applications folder and did a bit of renaming so it is now "Arduino" and the previous one is "Arduino-1.5-r2". But now, it doesn't even have all the boards the previous version had. I need to support the Adafruit Trinket Pro 5V (FTDI), and the other support, for the plain Trinket, the Trinket Pro 3V, and the Gemma would be nice. I've got a question out in the Adafruit forum regarding this, but I find that there is a lot of guesswork involved in trying to understand the platform.txt file and the boards.txt file. Is there a document anywhere that defines these files and how they all cohabit my development space?

supuflounder · 2015-03-14T01:32:44Z

Would it help at all if Arduino had an option to either write a log file as it is starting up, or produce a file which is its representation for boards, boot loaders, and platforms? Then I could post the file, and somebody could say "OH, the reason your definitions are not being seen is that..." instead of my having to do "black box" experiments trying to figure out what is going on. The document in the wiki is only of marginal help. I have spent three days trying to figure out how to program my Trinket Pro, with no success. In fact, I'm further behind now than I was; under the 1.5-r2 release, it could see the board types and compile, but the upload failed. Now, I can't even find the boards. The Adafruit support keeps putting me into the middle of circular link sequences, where each node has either incomplete or apparently conflicting information. I have tried every variant described in every link, and get NOTHING!

What I have done is take a zip file which contains boards.txt, bootloaders.txt, and platforms.txt, and installed it in

mylogin/documents/Arduino/hardware/avr

with no happiness. I tried moving them up to be just under "hardware" (siblings to the avr folder) and nothing changed. I have six projects waiting for me to be able to download code, and for the last six months, I could not compile some of the code (I'm hoping 1.6.2 really does fix these problems), and while I could download the code that did compile to an Arduino Uno for prototyping, I can't get it downloaded into the Trinket Pro in the "final product" implementation. This is extremely frustrating.

supuflounder · 2015-03-20T00:18:53Z

Using what I thought was the most recent version (1.6.2), I got the following error compiling my sketch:

[code]
/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/coan source -m -E -P -kb -DF_CPU=16000000L -DARDUINO=10602 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR /var/folders/qk/48tgmmgj3bq6x7_r2cxw_20w0000gn/T/sketch4171742237898923971.cpp
Using library Adafruit_NeoPixel in folder: /Users/flounder/Documents/Arduino/libraries/Adafruit_NeoPixel (legacy)
Using library colorref4 in folder: /Users/flounder/Documents/Arduino/libraries/colorref4 (legacy)
/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzn /var/folders/qk/48tgmmgj3bq6x7_r2cxw_20w0000gn/T/sketch5036631053244141579.cpp
processing.app.debug.RunnerException
at processing.app.debug.Compiler.compile(Compiler.java:359)
at processing.app.debug.Compiler.build(Compiler.java:98)
at processing.app.Sketch.build(Sketch.java:1169)
at processing.app.Sketch.build(Sketch.java:1142)
at processing.app.Editor$BuildHandler.run(Editor.java:1979)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException
at processing.app.preproc.CTagsParser.addPrototypes(CTagsParser.java:94)
at processing.app.preproc.CTagsParser.preprocess(CTagsParser.java:38)
at processing.app.preproc.CTagsBakedPreprocessor.parseCTagsOutput(CTagsBakedPreprocessor.java:113)
at processing.app.preproc.CTagsBakedPreprocessor.preprocess(CTagsBakedPreprocessor.java:52)
at processing.app.preproc.CTagsBakedPreprocessor.preprocess(CTagsBakedPreprocessor.java:42)
at processing.app.debug.Compiler.compile(Compiler.java:357)
... 5 more
[/code]
If you need the actual sketches to test this, I can send them.
joe

ffissore · 2015-03-23T13:51:57Z

@supuflounder yes please, send your sketch and the libraries it uses to f.fissore@arduino.cc

ArduinoBot · 2015-03-23T16:04:05Z

Build successful. Please test this code using one of the following:
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-232-linux32.tar.xz
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-232-linux64.tar.xz
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-232-windows.zip
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-232-macosx.zip
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-232-macosx-java-latest.zip

…tructs but we still don't deal with them

…ere the first function is defined and place the prototypes right before. Coan will replace excluded code with empty lines instead of removing them: this makes it easier to map preprocessed code with the original one

… are wrong though

…ts loaded first and children platform.txt can override its key value pairs

…mple when you have defined a custom hardware packge (in your own sketchbook/hardware folder) and you have not defined a platform.txt because your build.core is arduino:something

(defined in IDE/hardware/platform.txt) is used as a base for custom hardware (defined in SKETCHBOOK/hardware/platform.txt)

ArduinoBot · 2015-03-25T16:49:06Z

Build successful. Please test this code using one of the following:
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-233-linux32.tar.xz
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-233-linux64.tar.xz
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-233-windows.zip
http://arduino.cc/download.php?f=/javaide/pull_requests/arduino-PR-2729-BUILD-233-macosx.zip

NicoHood · 2015-04-30T09:00:23Z

With a custom core I get:
Cannot run program "{runtime.tools.avr-gcc.path}/bin/avr-g++": CreateProcess error=2, The system cannot find the file specified

May someone update this PR?

Without a custom core the bug, that drove me crazy is now gone. I really look forward to see this in the new IDE and working. An example where I need it:
My Infrared decoding library can use PinInterrupts and PinChangeinterrupts. 2nd thing needs a special library. If you include the PCINT library before the Infrared library (which is purely .h inline) it can detect the inclusion and a) include the library as well and dynamically add those new pins that are available to decode the signals. But right now the definition is not passed to the Infrared library because it somehow gets reordered wrong.

This PR fixes it, but it doesnt like my HID-Project core. You might want to try it yourself.

RobAtticus · 2015-07-13T20:36:32Z

Hi,

This issue seems to have gone stale a bit, but I was wondering what the outlook for this is. These changes seem greatly useful to me and wanted to know if I can expect to see them hit the mainline sometime soon.

Jeppedy · 2015-08-17T14:41:12Z

Would love to see the preprocessor fixes merged. It has taken a good deal of my time before I figured out what was odd about the Arduino IDE. I imagine others would love being saved from this confusion about IDE behavior...

NicoHood · 2015-08-17T14:45:22Z

Now there are conflicts in the branch...
I aggree to jeppedy. Please rebuild the IDE with the bot then to test it again.

ffissore · 2015-09-11T12:21:56Z

Superseded by #3779

ffissore mentioned this pull request Mar 4, 2015

New preprocessor #2636

Closed

This was referenced Mar 12, 2015

Compilation error #2746

Closed

Source modifications lead to compile failures #2755

Closed

Compilation error: missing terminating " character #2762

Closed

ffissore mentioned this pull request Mar 16, 2015

Compile error when declaring a pointer-function #2772

Closed

PaulStoffregen mentioned this pull request Mar 19, 2015

Add library dependency detection #2792

Closed

ffissore mentioned this pull request Mar 23, 2015

wrong placement of arduino.h #1310

Closed

Federico Fissore added 10 commits March 25, 2015 15:05

Introducing new preprocessor, based on coan2 and ctags

187db5d

New preprocessor: removed useless AbstractPreprocessor class

db15c22

Added ignored ctags parser test: we now how to call ctags to output s…

b551305

…tructs but we still don't deal with them

Updated coan binary for mac, new zip uploaded

116fef3

Added support to sketches with multiple files: #line preproc commands…

eea7a24

… are wrong though

Fixed #line offset

8876de3

Added compile_every_example.sh script

8d43a67

Moving coan and ctags definitions into a parent platform.txt: this ge…

789d18d

…ts loaded first and children platform.txt can override its key value pairs

ctags and coan: fixed failing tests

a814ac3

Federico Fissore added 5 commits March 25, 2015 15:05

TargetPlatform.getTool() does not guarantee to return a tool, for exa…

ea5f8f7

…mple when you have defined a custom hardware packge (in your own sketchbook/hardware folder) and you have not defined a platform.txt because your build.core is arduino:something

Windows: fixed command line arguments quoting

d633d5f

When loading sketchbook defined hardware, generic platform.txt

b65f986

(defined in IDE/hardware/platform.txt) is used as a base for custom hardware (defined in SKETCHBOOK/hardware/platform.txt)

Fixed properties loading order

71826c0

Default args values are removed from the prototype. See #2729 (comment)

b1de253

cmaglie added feature request A request to make an enhancement (not a bug fix) Component: Compilation Related to compilation of Arduino sketches labels Apr 15, 2015

ffissore mentioned this pull request Apr 24, 2015

Arduino throws StackOverflow exception for long strings #2985

Closed

ffissore self-assigned this May 12, 2015

PaulStoffregen force-pushed the master branch from 57dcf07 to 5a68e34 Compare May 19, 2015 17:51

This was referenced May 28, 2015

Fix for single line comments ending in \ #3240

Closed

Error message line numbers incorrect when compiling multiple .ino files #1479

Closed

ffissore mentioned this pull request Sep 11, 2015

A new compiler #3779

Merged

ffissore closed this Sep 11, 2015

ffissore modified the milestone: Release 1.6.6 Sep 11, 2015

ffissore deleted the coanctags branch September 14, 2015 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New preprocessor #2729

New preprocessor #2729

ffissore commented Mar 4, 2015

Wackerbarth commented Mar 9, 2015

ffissore commented Mar 9, 2015

mastrolinux commented Mar 9, 2015

mastrolinux commented Mar 9, 2015

ffissore commented Mar 10, 2015

ffissore commented Mar 10, 2015

ArduinoBot commented Mar 10, 2015

ffissore commented Mar 13, 2015

supuflounder commented Mar 13, 2015

supuflounder commented Mar 14, 2015

supuflounder commented Mar 20, 2015

ffissore commented Mar 23, 2015

ArduinoBot commented Mar 23, 2015

ArduinoBot commented Mar 25, 2015

NicoHood commented Apr 30, 2015

RobAtticus commented Jul 13, 2015

Jeppedy commented Aug 17, 2015

NicoHood commented Aug 17, 2015

ffissore commented Sep 11, 2015

New preprocessor #2729

New preprocessor #2729

Conversation

ffissore commented Mar 4, 2015

Wackerbarth commented Mar 9, 2015

ffissore commented Mar 9, 2015

mastrolinux commented Mar 9, 2015

mastrolinux commented Mar 9, 2015

ffissore commented Mar 10, 2015

ffissore commented Mar 10, 2015

ArduinoBot commented Mar 10, 2015

ffissore commented Mar 13, 2015

supuflounder commented Mar 13, 2015

supuflounder commented Mar 14, 2015

supuflounder commented Mar 20, 2015

ffissore commented Mar 23, 2015

ArduinoBot commented Mar 23, 2015

ArduinoBot commented Mar 25, 2015

NicoHood commented Apr 30, 2015

RobAtticus commented Jul 13, 2015

Jeppedy commented Aug 17, 2015

NicoHood commented Aug 17, 2015

ffissore commented Sep 11, 2015