Signed-off-by: Nazarii Hnydyn <nazariig@mellanox.com>

* [Mellanox] Fix 'sensors.conf' mapping for SN4700

Signed-off-by: Shlomi Bitton <shlomibi@mellanox.com>

* Fix some labels name

…0 and Z9100 (sonic-net#4593)

For detecting transceiver change events through xcvrd in DellEMC S6000, S6100 and Z9100 platforms.

- In S6000, rename 'get_transceiver_change_event' in chassis.py to 'get_change_event' and return appropriate values.
- In S6100, implement 'get_change_event' through polling method (poll interval = 1 second) in chassis.py (Transceiver insertion/removal does not generate interrupts due to a CPLD bug)
- In Z9100, implement 'get_change_event' through interrupt method using select.epoll().

…nic-net#4652)

…#4688)

201911 cel dev

… packets (sonic-net#4412)

Modified caclmgrd behavior to enhance control plane security as follows:

Upon starting or receiving notification of ACL table/rule changes in Config DB:
1. Add iptables/ip6tables commands to allow all incoming packets from established TCP sessions or new TCP sessions which are related to established TCP sessions
2. Add iptables/ip6tables commands to allow bidirectional ICMPv4 ping and traceroute
3. Add iptables/ip6tables commands to allow bidirectional ICMPv6 ping and traceroute
4. Add iptables/ip6tables commands to allow all incoming Neighbor Discovery Protocol (NDP) NS/NA/RS/RA messages
5. Add iptables/ip6tables commands to allow all incoming IPv4 DHCP packets
6. Add iptables/ip6tables commands to allow all incoming IPv6 DHCP packets
7. Add iptables/ip6tables commands to allow all incoming BGP traffic
8. Add iptables/ip6tables commands for all ACL rules for recognized services (currently SSH, SNMP, NTP)
9. For all services which we did not find configured ACL rules, add iptables/ip6tables commands to allow all incoming packets for those services (allows the device to accept SSH connections before the device is configured)
10. Add iptables rules to drop all packets destined for loopback interface IP addresses
11. Add iptables rules to drop all packets destined for management interface IP addresses
12. Add iptables rules to drop all packets destined for point-to-point interface IP addresses
13. Add iptables rules to drop all packets destined for our VLAN interface gateway IP addresses
14. Add iptables/ip6tables commands to allow all incoming packets with TTL of 0 or 1 (This allows the device to respond to tools like tcptraceroute)
15. If we found control plane ACLs in the configuration and applied them, we lastly add iptables/ip6tables commands to drop all other incoming packets

… present (sonic-net#4581)

Since the introduction of VRF, interface-related tables in ConfigDB will have multiple entries, one of which only contains the interface name and no IP prefix. Thus, when iterating over the keys in the tables, we need to ignore the entries which do not contain IP prefixes.

…ic-net#4672)

both for sonic-slave-stretch and sonic-slave-buster

I found that with IPv4Network types, calling list(ip_ntwrk.hosts()) is reliable. However, when doing the same with an IPv6Network, I found that the conversion to a list can hang indefinitely. This appears to me to be a bug in the ipaddress.IPv6Network implementation. However, I could not find any other reports on the web.

This patch changes the behavior to call next() on the ip_ntwrk.hosts() generator instead, which returns the IP address of the first host.

…et#4670)

To support t0 config

Auto restart restapi server after cert rollover

… fail (sonic-net#4675)

This change allows the recursive `git clean` and `git reset` commands to continue even if they encounter an error in one of the submodules. Previously, if an error was encountered, the operation would terminate with a message similar to the following:

Stopping at 'src/sonic-mgmt-framework'; script returned non-zero status.

 [aclorch] Add support for creating ingress and egress MIRROR tables
 concurrently (sonic-net#1286)
[proxy_arp] Implement proxy ARP feature (sonic-net#1302)
Fix LAG member test case (sonic-net#1304)
[orchagent] Set default MTU for the underlay loopback interface (sonic-net#1299)

  Fix memory leak in pyext when Selectable is returned to Python (sonic-net#343)

Make sure db_migrator is run after all config are loaded during (sonic-net#926)
Vnet alias mapping (sonic-net#924)
Changes to make lldp show command for multi-npu platforms. (sonic-net#914)
[Mellanox] Fix thermal control issue: use natural sort for fan
status and thermal status (sonic-net#836)
[Mellanox] add document for thermal control related cli (sonic-net#832)

…atches (sonic-net#4696)

Signed-off-by: Volodymyr Samotiy <volodymyrs@mellanox.com>

 Corrected the copp rule as per NAT HLD (sonic-net#1300)

…net#4677)

Backport thermal control feature from master branch to 201911 branch by cherry-picking commits and manually resolving conflicts.

…onic-net#4752)

**- Why I did it**

- Skip thermalctld in DellEMC S6000, S6100, Z9100 and Z9264 platforms.
- Change the return type of thermal Platform APIs in DellEMC S6000, S6100 and Z9100 platforms to 'float'.

**- How I did it**

- Add 'skip_thermalctld:true' in pmon_daemon_control.json for DellEMC S6000, S6100, Z9100 and Z9264 platforms.
- Made changes in thermal.py, for 'get_temperature', 'get_high_threshold' and 'get_low_threshold' to return 'float' value.

**- How to verify it**

- Check thermalctld is not running in 'pmon'.
- Wrote a python script to load Chassis class and then call the APIs accordingly and verify the return type.

Signed-off-by: Nazarii Hnydyn <nazariig@mellanox.com>

…#4743)

* [Mellanox] Initialize system LED color to green for 201911

* Rename variable to make it more readable

…tem (sonic-net#4431)

* Run fsck filesystem check support prior mounting filesystem

If the filesystem become non clean ("dirty"), SONiC does not run fsck to
repair and mark it as clean again.

This patch adds the functionality to run fsck on each boot, prior to the
filesystem being mounted. This allows the filesystem to be repaired if
needed.

Note that if the filesystem is maked as clean, fsck does nothing and simply
return so this is perfectly fine to call fsck every time prior to mount the
filesystem.

How to verify this patch (using bash):

Using an image without this patch:

Make the filesystem "dirty" (not clean)
[we are making the assumption that filesystem is stored in /dev/sda3 - Please adjust depending of the platform]
[do this only on a test platform!]

dd if=/dev/sda3 of=superblock bs=1 count=2048
printf "$(printf '\\x%02X' 2)" | dd of="superblock" bs=1 seek=1082 count=1 conv=notrunc &> /dev/null
dd of=/dev/sda3 if=superblock bs=1 count=2048

Verify that filesystem is not clean
tune2fs -l /dev/sda3 | grep "Filesystem state:"

reboot and verify that the filesystem is still not clean
Redo the same test with an image with this patch, and verify that at next reboot the filesystem is repaired and becomes clean.

fsck log is stored on syslog, using the string FSCK as markup.

…onic-net#4673)

This will allow us to disable services and have it persist across reboots by using the `systemctl mask` operation

Fix hostcfgd so that changes to the "FEATURE" table in ConfigDB are properly handled. Three changes here:

1. Fix indenting such that the handling of each key actually occurs in the for key in status_data.keys(): loop
2. Add calls to sudo systemctl mask and sudo systemctl unmask as appropriate to ensure changes persist across reboots
3. Substitute returns with continues so that even if one service fails, we still try to handle the others

Note that the masking is persistent, even if the configuration is not saved. We may want to consider only calling systemctl enable/disable in hostcfgd when the DB table changes, and only call systemctl mask/unmask upon calling config save.

Found another syncd timing issue related to clock going backwards.
To be safe disable the ntp long jump.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

**- Why I did it**

For decoding system EEPROM of S6000 based on Dell offset format and S6000-ON’s system EEPROM in ONIE TLV format.

**- How I did it**

- Differentiate between S6000 and S6000-ON using the product name available in ‘dmi’  ( “/sys/class/dmi/id/product_name” )
- For decoding S6000 system EEPROM in Dell offset format and updating the redis DB with the EEPROM contents, added a new class ‘EepromS6000’ in eeprom.py, 
- Renamed certain methods in both Eeprom, EepromS6000 classes to accommodate the plugin-specific methods.

**- How to verify it**

- Use 'decode-syseeprom' command to list the system EEPROM details.
- Wrote a python script to load chassis class and call the appropriate methods.

UT Logs: [S6000_eeprom_logs.txt](https://github.com/Azure/sonic-buildimage/files/4735515/S6000_eeprom_logs.txt), [S6000-ON_eeprom_logs.txt](https://github.com/Azure/sonic-buildimage/files/4735461/S6000-ON_eeprom_logs.txt)
Test script: [eeprom_test_py.txt](https://github.com/Azure/sonic-buildimage/files/4735509/eeprom_test_py.txt)

Signed-off-by: Shlomi Bitton <shlomibi@mellanox.com>

**- Why I did it**
To ensure telemetry service is enabled by default after installing a fresh SONiC image

**- How I did it**
Set telemetry feature status to "enabled" when generating init_cfg.json file

…onic-net#4767)

* fix redistribution connected for ipv6 also

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

Switching to Flannel from Calico which brings down the image size by around 500+MB.

…onic-net#4757)

)

**- Why I did it**
When I tested auto-restart feature of swss container by manually killing one of critical processes in it, swss will be stopped. Then syncd container as the peer container should also be
stopped as expected. However, I found sometimes syncd container can be stopped, sometimes
it can not be stopped. The reason why syncd container can not be stopped is the process
(/usr/local/bin/syncd.sh stop) to execute the stop() function will be stuck between the lines 164 –167. Systemd will wait for 90 seconds and then kill this process.

164 # wait until syncd quit gracefully
165 while docker top syncd$DEV | grep -q /usr/bin/syncd; do
166 sleep 0.1
167 done

The first thing I did is to profile how long this while loop will spin if syncd container can be
normally stopped after swss container is stopped. The result is 5 seconds or 6 seconds. If syncd
container can be normally stopped, two messages will be written into syslog:

str-a7050-acs-3 NOTICE syncd#dsserve: child /usr/bin/syncd exited status: 134
str-a7050-acs-3 INFO syncd#supervisord: syncd [5] child /usr/bin/syncd exited status: 134

The second thing I did was to add a timer in the condition of while loop to ensure this while loop will be forced to exit after 20 seconds:

After that, the testing result is that syncd container can be normally stopped if swss is stopped
first. One more thing I want to mention is that if syncd container is stopped during 5 seconds or 6 seconds, then the two log messages can be still seen in syslog. However, if the execution 
time of while loop is longer than 20 seconds and is forced to exit, although syncd container can be stopped, I did not see these two messages in syslog. Further, although I observed the auto-restart feature of swss container can work correctly right now, I can not make sure the issue which syncd container can not stopped will occur in future.

**- How I did it**
I added a timer around the while loop in stop() function. This while loop will exit after spinning
20 seconds.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

 [201911][thermal control] Backport changes from master branch (sonic-net#929)
     [201911][config] Support abbreviation (sonic-net#933)
       Add 'hw-management-generate-dump.sh' to 'show techsupport'
       command (sonic-net#934)
       [fwutil]: Update fwutil to v2.0.0.0. (sonic-net#942)
       Fixes bug for PFCWD feature parameters (sonic-net#838)
     Fixed fast-reboot for BFN platform (sonic-net#871)
     [config] Add 'interface transceiver' subgroup with 'lpmode' and
     'reset' subcommands (sonic-net#904)
      [warm-reboot]: added pre-check for ISSU file (sonic-net#915)
       [config] Don't attempt to restart disabled services (sonic-net#944)

Revert "[portsorch] Enable port-level buffer drop counters (sonic-net#1237)"
(sonic-net#1308) Add more log message, fix test code (sonic-net#1239)

[syseepromd] Prevent the syseepromd from termination (#56)
 [thermalctld] Fix invalid warning status (#58)

 Add missed BGP tables into the schema (sonic-net#351)

[eeprom] Add try-except to catch the IOError (#85)
[sfputilbase.py] Don't try to print EEPROM sysfs file name if we failed
 to read from it (#81)

…encing (sonic-net#4788)

* Fix the Build on 201911 (Stretch) where the directory
/usr/lib/systemd/system/ does not exist so creating
manually. Change should not harm Master (buster) where
the directory is created by Linux

* Fix as per review comments

[syncd] Use steady clock for TimerWatchdog (sonic-net#613)

Signed-off-by: Nazarii Hnydyn <nazariig@mellanox.com>

…c-net#4705)

* Update to sonic_cfggen and utilities to populate the (pci) device id in the "asic_id" field in the DEVICE_METADATA. This Id is parsed from the file "asic.conf" file in the device/<platform> dir. The format of entries are  eg:  for a 2 ASIC platform.
DEV_ID_ASIC_0=03:00.0
DEV_ID_ASIC_1=04:00.0
 
Going forward will use this device id as the asic instance ID passed to syncd/sai while doing create_switch. Current support is limited, supports only one TD2 platform.

Images built from 201911 branch and installed on devices where we mount /var/lib/docker in RAM (because the HDD is small) were failing as there was not enough space to untar docker.tar.gz . This is due to the increase in total number of containers in the image.

As of today, /var/lib/docker contains 1.1 GB of data. Therefore, this PR increases the size of the ramdisk to 1.3 GB to accommodate all the containers as of now and any new container going forward.
Example output below from an Arista-7050-QX32 SKU:

admin@str-a7050-acs-2:~$ df -h
Filesystem Size Used Avail Use% Mounted on
.....
tmpfs 1.3G 1.1G 221M 84% /var/lib/docker
.....
Verified all docker running fine and interfaces/bgp are up.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

…#4804)

to other packages so intsalling them explicitly.

lldp: For multi-npu platforms make sure to add Backplane Interface also
as Interface Match List.

… provided (sonic-net#4787)

Don't limit iptables connection tracking to TCP protocol; allow connection tracking for all protocols. This allows services like NTP, which is UDP-based, to receive replies from an NTP server even if the port is blocked, as long as it is in reply to a request sent from the device itself.

While migrating to SONiC 20181130, identified a couple of issues:
1. union-mount needs /host/machine.conf parameters for vendor specific checks : however, in case of migration, the /host/machine.conf is extracted from ONIE only in https://github.com/Azure/sonic-buildimage/blob/master/files/image_config/platform/rc.local#L127. 
2. Since grub.cfg is updated to have net.ifnames=0 biosdevname=0, 70-persistent-net.rules changes are no longer required.

Fix for command. show interface transceiver eeprom -d Ethernet (sonic-net#955)
Updated natshow script to support DNAT Pool changes (sonic-net#921)

* added files to create SKU Mellanox-SN3800-D28C50

Co-authored-by: Madhan Babu <madhan@arc-build-server.mtr.labs.mlnx>

* modified files relevant to SKU Mellanox-SN3800-D112C8

Co-authored-by: Madhan Babu <madhan@arc-build-server.mtr.labs.mlnx>

* added files to create SKU Mellanox-SN3800-C64
Co-authored-by: Madhan Babu <madhan@arc-build-server.mtr.labs.mlnx>

Co-authored-by: Madhan Babu <madhan@arc-build-server.mtr.labs.mlnx>

Backport of sonic-net#4831 to the 201911 branch

…rm. (sonic-net#4859)

Verified after loading on TH platforms cpu usage gone down:

Previous:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2521 root 20 0 1512860 360452 63540 S 144.6 4.4 8:00.03 syncd

After Fix:
7500 root 20 0 1592420 350912 64184 S 45.4 4.3 3:50.99 syncd

Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>

* added files to create SKU Mellanox-SN3800-D24C52

Co-authored-by: Madhan Babu <madhan@arc-build-server.mtr.labs.mlnx>

…#4781)

* Change port index in port_config.ini to 1-based
* Add default port index to port_config.ini, change platform plugins to accept 1-based port index
* fix port index in sfp_event.py

…ic-net#4790)

…vice (sonic-net#3690)

* [platform/cel]: add fancontrol service support for dx010

* [device/celestica]: add hysteresis temp to dx010 fancontrol configuration

FDB/ARP/Default routes files are deleted after swssconfig. This
makes debugging/validation of device conversion hard. This PR
saves those files in order to facilitate debugging of device conversion.

signed-off-by: Tamer Ahmed <tamer.ahmed@microsoft.com>

…t#4714)

- Ensure all features (services) are in the configured state when hostcfgd starts
- Better functionalization of code
- Also replace calls to deprecated `has_key()` method in `tacacs_server_handler()` and `tacacs_global_handler()` with `in` keyword.

This PR depends on sonic-net/sonic-utilities#944, otherwise `config load_minigraph` will fail when trying to restart disabled services.

When building the SONiC image, used systemd to mask all services which are set to "disabled" in init_cfg.json.

This PR depends on sonic-net/sonic-utilities#944, otherwise `config load_minigraph will fail when trying to restart disabled services.

Issue: Port with AOC cable does not come up when "sfputil reset <port_name>" is executed.

Modified the incorrect mask used in reset API to resolve the issue.

sonic-net#4706)

**- Why I did it**
After discussed with Joe, we use the string "/usr/bin/syncd\s" in Monit configuration file to monitor 
syncd process on Broadcom and Mellanox. Due to my careless, I did not find this bug during the 
previous testing. If we use the string "/usr/bin/syncd" in Monit configuration file to monitor the 
syncd process, Monit will not detect whether syncd process is running or not. 

If we ran the command  `sudo monit procmactch “/usr/bin/syncd”` on Broadcom, there will be three 
processes in syncd container which matched this "/usr/bin/syncd": `/bin/bash /usr/bin/syncd.sh
wait`, `/usr/bin/dsserve /usr/bin/syncd –diag -u -p /etc/sai.d/sai.profile` and `/usr/bin/syncd –diag -
u -p /etc/sai.d/said.profile`. Monit will select the processes with the highest uptime (at there 
`/bin/bash /usr/bin/syncd.sh wait`) to match and did not select `/usr/bin/syncd –diag -u -p
/etc/sai.d/said.profile` to match. 

Similarly, On Mellanox Monit will also select the process with the highest uptime (at there 
`/bin/bash /usr/bin/syncd.sh wait`) to match and did not select `/usr/bin/syncd –diag -u -p
/etc/sai.d/said.profile` to match.

That is why Monit is unable to detect whether syncd process is running or not if we use the string “/usr/bin/syncd” in Monit configuration file. If we use the string "/usr/bin/syncd\s" in Monit configuration file, Monit can filter out the process `/bin/bash /usr/bin/syncd.sh wait` and thus can correctly monitor the syncd process.

**- How I did it**

**- How to verify it**

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

…cesses file (sonic-net#4863)

The program name in critical_processes file must match the program name defined in supervisord.conf file.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>

…ner file (sonic-net#4891)

While testing reboot case for 201911 facing error:
supervisor-proc-exit-listener FATAL command at '/usr/bin/supervisor-proc-exit-listener' is not executable

Signed-off-by: Roman Savchuk <romanx.savchuk@intel.com>

thermalctld throwing error messages because it is not yet fully configured, disabling it for now on arista platforms.

Co-authored-by: Zhi Yuan Carl Zhao <zyzhao@arista.com>

Add changes for syslog support for containers running in namespaces on multi ASIC platforms.
On Multi ASIC platforms

Rsyslog service is only running on the host. There is no rsyslog service running in each namespace.
On multi ASIC platforms the rsyslog service on the host will be listening on the docker0 ip address instead of loopback address.
The rsyslog.conf on the containers is modified to have omfwd target ip to be docker0 ipaddress instead of loopback ip

Signed-off-by: Arvindsrinivasan Lakshmi Narasimhan <arlakshm@microsoft.com>

…rms (sonic-net#4774)

* Changes to make default route programming
correct in multi-asic platform where frr is not running
in host namespace. Change is to set correct administrative distance.
Also make NAMESPACE* enviroment variable available for all dockers
so that it can be used when needed.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Fix review comments

* Review comment to check to add default route
only if default route exist and delete is successful.

…onic-net#4820)

* [systemd-generator]: Fix the code to make sure that dependencies
of host services are generated correctly for multi-asic platforms.
Add code to make sure that systemd timer files are also modified
to add the correct service dependency for multi-asic platforms.

Signed-off-by: SuvarnaMeenakshi <sumeenak@microsoft.com>

* [systemd-generator]: Minor fix, remove debug code and
remove unused variable.

…platform. (sonic-net#4779)

* Support for connecting to DB in namespace via IP:port ( using docker bridge network ) for applications in multi-asic platform.

* Added the default IP as 127.0.0.1 if the IPaddress derivation from interface fails.
Moved the localhost loopback IP binding logic into the supervisor.j2 file.

… both host and docker namespace (sonic-net#4838)

* [sonic-buildimage] Changes to make network specific sysctl
common for both host and docker namespace (in multi-npu).

This change is triggered with issue found in multi-npu platforms
where in docker namespace
net.ipv6.conf.all.forwarding was 0 (should be 1) because of
which RS/RA message were triggered and link-local router were learnt.

Beside this there were some other sysctl.net.ipv6* params whose value
in docker namespace is not same as host namespace.

So to make we are always in sync in host and docker namespace
created common file that list all sysctl.net.* params and used
both by host and docker namespace. Any change will get applied
to both namespace.

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

* Address Review Comments and made sure to invoke augtool
only one and do string concatenation of all set commands

* Address Review Comments.

* Tests of FRR templates which rendered by sonic-cfggen

- Xilinx/pericom peripherals are not actively used in DellEMC S6100 switch.
- These peripherals are throwing PCIE corrected messages in some of the units and filling syslog.
- Since it is not usable disabling it at startup.

…is no one (sonic-net#4860)

If some table with a list of tuples (interface name, ip prefix) has ip prefixes without a mask length, it will cause issues in SONiC. For example quagga and frr will treat ipv4 address without a mask, so "10.20.30.40" address will be treated as "10.0.0.0/8", which is dangerous.

The fix here is that when pfx_filter get a tuple (interface name, ip prefix), where the ip prefix doesn't have prefix mask length, add a mask by default: "/32 for ipv4 addresses, /128 for ipv6 addresses".

Co-authored-by: Pavel Shirshov <pavel.contrib@gmail.com>

Signed-off-by: Shlomi Bitton <shlomibi@mellanox.com>

* Tests for bgpcfgd templates

…-net#3868)

In the scope of migration from docker shell plugin to docker connection plugin, we need to have docker-ce-cli installed in docker-sonic-mgmt. sonic-net/sonic-mgmt#1269

Added docker-ce-cli package to docker-sonic-mgmt.

…ic-net#3978)

* Install python-subnettree to sonic-mgmt container.

…ild break

looks like version 0.42 has build issues

Signed-off-by: Guohan Lu <gulv@microsoft.com>

…c-net#4110)

The snmp tool is required for interacting with certain type
of PDU hosts in platform PSU/power related testing. This
change is to have the snmp tool pre-built in the sonic-mgmt
docker image.

Signed-off-by: Xin Wang <xinw@mellanox.com>

…net#4325)

. env-201811/bin/activate

The ansible 2.0.0.2 will be used.