Sierra Wireless EM7345 on Debian Linux

 Hardware, Kernel and drivers, Linux, Network  Comments Off on Sierra Wireless EM7345 on Debian Linux
May 132018
 

Hardware: Lenovo ThinkPad W550s with Sierra Wireless EM7345

Software: Debian 10 (buster/testing)

Operator: Sonera (Finland)

After Debian installation the modem is recognized, but somehow a working connection just can’t be created:

# mmcli -L

Found 1 modems:
 /org/freedesktop/ModemManager1/Modem/0 [Sierra Wireless Inc.] Sierra Wireless EM7345 4G LTE

# mmcli -m 0

/org/freedesktop/ModemManager1/Modem/0 (device id '<snip>')
-------------------------
Hardware | manufacturer: 'Sierra Wireless Inc.'
| model: 'Sierra Wireless EM7345 4G LTE'
| revision: 'V1.1,11'
| H/W revision: 'unknown'
| supported: 'gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: '<snip>'
-------------------------
System | device: '/sys/devices/pci0000:00/0000:00:14.0/usb2/2-4'
| drivers: 'cdc_acm, cdc_ncm'
| plugin: 'Generic'
| primary port: 'ttyACM0'
| ports: 'enx000011121314 (net), ttyACM0 (at), ttyACM2 (at)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'none'
| unlock retries: 'sim-pin (3), sim-pin2 (3), sim-puk (10), sim-puk2 (10)'
| state: 'registered'
| power state: 'on'
| access tech: 'lte'
| signal quality: '35' (recent)
-------------------------
Modes | supported: 'allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: 2g, 3g, 4g; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
3GPP | imei: '<snip>'
| enabled locks: 'none'
| operator id: '24491'
| operator name: 'FI SONERA'
| subscription: 'unknown'
| registration: 'home'
| EPS UE mode: 'csps-2'
-------------------------
SIM | path: '/org/freedesktop/ModemManager1/SIM/0'

-------------------------
Bearers | paths: '/org/freedesktop/ModemManager1/Bearer/0'

# nmcli connection add type gsm ifname "" con-name 4G apn internet connection.autoconnect no
Connection '4G' (ef2b6f2d-e072-44cf-a9f8-831b79b179f2) successfully added.

# nmcli connection up 4G
Error: Connection activation failed: Unknown error

daemon.log revealed that modem got stuck on “Connect: ppp0 <–> /dev/ttyACM0” and a bit later on “Couldn’t initialize PDP context with our APN: ‘Serial command timed out'”

Problem seems to be that NCM -mode isn’t actually supported by EM7345 – even though advertised by the firmware (and H/W revision says unknown). Small confirmation for this theory is that Lenovo provides drivers for older Windows versions where MBIM isn’t natively supported.

Let’s override defaults for cdc_ncm and prefer MBIM -mode.

# cat /sys/module/cdc_ncm/parameters/prefer_mbim
N
# echo "options cdc_ncm prefer_mbim=Y" > /etc/modprobe.d/cdc_ncm.conf
# modprobe -r cdc_mbim cdc_ncm
# modprobe cdc_mbim
# cat /sys/module/cdc_ncm/parameters/prefer_mbim
Y
# systemctl restart ModemManager
# mmcli -L

Found 1 modems:
 /org/freedesktop/ModemManager1/Modem/0 [Sierra Wireless Inc.] MBIM [1199:A001]

# mmcli -m 0

/org/freedesktop/ModemManager1/Modem/0 (device id '<snip>')
-------------------------
Hardware | manufacturer: 'Sierra Wireless Inc.'
| model: 'MBIM [1199:A001]'
| revision: 'FIH7160_V1.2_WW_01.1616.01'
| H/W revision: 'XMM7160_V1.2_MBIM_GNSS_NAND_RE'
| supported: 'gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: '<snip>'
-------------------------
System | device: '/sys/devices/pci0000:00/0000:00:14.0/usb2/2-4'
| drivers: 'cdc_acm, cdc_mbim'
| plugin: 'Sierra'
| primary port: 'cdc-wdm0'
| ports: 'wwp0s20u4 (net), cdc-wdm0 (mbim), ttyACM0 (at), ttyACM2 (at)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'none'
| unlock retries: 'sim-pin2 (3)'
| state: 'registered'
| power state: 'on'
| access tech: 'lte'
| signal quality: '35' (recent)
-------------------------
Modes | supported: 'allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: 2g, 3g, 4g; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
3GPP | imei: '<snip>'
| enabled locks: 'fixed-dialing'
| operator id: '24491'
| operator name: 'FI SONERA'
| subscription: 'unknown'
| registration: 'home'
| EPS UE mode: 'csps-2'
-------------------------
SIM | path: '/org/freedesktop/ModemManager1/SIM/0'

-------------------------
Bearers | paths: 'none'

# nmcli connection up 4G
Error: Timeout expired (90 seconds)

…but still not quite there.
This time daemon.log says that the modem was stuck in 3GPP Registration state loop, idle -> registering -> home -> idle.

Reason for this behavior is Sonera, which (still) doesn’t support IPv6, and somehow fails the handshake if IP type ipv4v6 is offered first. Debug was done by running simple-connect manually:

# systemctl restart ModemManager
# mmcli -m 0 --simple-connect="apn=internet,ip-type=ipv4"
successfully connected the modem
# mmcli -m 0 --simple-disconnect
successfully disconnected all bearers in the modem
# systemctl restart ModemManager
# mmcli -m 0 --simple-connect="apn=internet,ip-type=ipv4v6"
error: couldn't connect the modem: 'Timeout was reached'
# systemctl restart ModemManager
# mmcli -m 0 --simple-connect="apn=internet,ip-type=ipv4"
successfully connected the modem
# mmcli -m 0 --simple-disconnect
successfully disconnected all bearers in the modem

So, how to translate this to network-manager actually handling the connection? Luckily just ignoring IPv6 altogether changes ip-type -setting, so (with nmcli) this is quite easily fixable:

# nmcli connection modify 4G ipv6.method ignore
# nmcli connection up 4G
Connection successfully activated (D-Bus active path: /<snip>)

…and finally, a working connection.

Getting more speed out of OpenWRT-enabled TL-WR1043NDv2

 Hardware, Kernel and drivers, Linux, Network  Comments Off on Getting more speed out of OpenWRT-enabled TL-WR1043NDv2
Mar 202018
 

Basic setup and debug

1Gbps/100Mbps -cable connection, router, couple of cat6 cables, laptop & speedtest.net (note: speedtest.net might not give reliable results on high-speed links).

 

Default speeds (without a router).
Cable modem is setup as a dumb bridge, it’ll just be a media converter between ethernet and the DOCSIS 3.1 -world. First tests were run with laptop hooked up to it with cat6 -cable, public IP-address resided on the laptop and “Hey, this works”. Plugged it into existing infrastructure (WDR4300) and carried on with my normal life.

Then came the next weekend and I had a bit too much time in hand to benchmark – and I found out that the wireless router runs out CPU if I download something big and the speed was capped at about 240Mbps…

What if I’ll change a router?
WDR4300 has an AR9344 CPU running at 560MHz – my reserve WR1043NDv2 has QCA9558 CPU running at 720MHz – a whopping 160MHz (~28.5%) more! I took a look at OpenWRT OpenSSL benchmarks and they confirmed my theory, CPU should be about a quarter faster, but would it translate into real world performance?

This is a bit like comparing apples and oranges, because the CPUs are different, firewall rules are close but not identical, and the latter one doesn’t handle wireless at the same time, but the speed difference is still noticeable.

What if I’ll simplify firewall rules?
Replaced all the default LEDE -firewall rules with just one NAT-rule, just allowed everything, and … about +10% more bandwidth.
How do I get more speed out of a CPU? Overclock!
Had a crazy idea, is it possible to overclock a router? And found out it actually is possible, just backup mtd0 (u-boot) -partition, replace default MHz -values with hex-editor to something else and burn the image back to router. Or use pre-made images or some chinese guys closed-source bootloader.

720MHz to 1GHz (+38%) and +32% more bandwidth with default firewall, nice!

Now let’s try it with a bare minimum firewall…
Same as before, just one NAT-rule.

+11% – and the speed is almost doubled from the start (WDR4300 and ~240Mbps).

What is this “NAT Boost / Hardware NAT” -feature in stock TP-Link firmware?
Back to default CPU speeds, stock firmware has some kind of shortcut for packets going through a NAT. Let’s see if we have..yes, OpenWRT is bringing SFE (Shortcut Forwarding Engine) -support to their later release.

Luckily there were some images ready for testing.

Shortcut forwarding engine, with just a NAT-rule.
Let’s combine that with a minimal firewall setup, just a single NAT-rule.
Shortcut forwarding engine, overclocked.
And what happens if we push the CPU from 720MHz to 1GHz. (With default LEDE-firewall.)
Full speed!
Let’s drop the firewall (use just a single NAT-rule), what’s the maximum attainable speed?

…not bad, just about 3.5 times faster than the original, and just a bit short of “full wirespeed”. But I think I’ll stick with a firewalled version and take that ~80Mbps hit.

Jul 012017
 
Suomenkielinen versio täällä.

Why?

One day I just needed an access control system and reservation calendar to a common use sauna, and it had an electric lock (24VAC) already in place.

Hardware (links to Finnish shops):

Stage 1 (memory card):

Raspberry Pi got the newest Raspbian Jessie, which was at the moment in version 2017-06-21. Kernel for the distribution is 4.9.28-v7+.

Stage 2 (assembly):

  • detach controller from the TFT (TFT-enclosure required this)
  • assemble the TFT enclosure
  • attach the controller to the TFT, make sure both cables are in place
  • drill holes to the bottom of RasPi enclosure (6mm drill), to use TFT -attachment points
  • insert memory card to RasPi
  • attach RasPi with it’s enclosure to TFT
  • attach the cable from TFT to RasPi, along with 5V(pin4) and Gnd(pin6) with jumper wires
  • attach NFC-reader with jumper wires to RasPi, you’ll use pins 1,3,5,11,12,15,16,19,21-26 (f.ex 3.3V, Gnd, i2c & SPI)
    (Explore NFC Board Schematic)
  • attach relay card module to RasPi, relay needs ~80mAh of current so take note of this when deciding the PSU
    pins 2 (5V -> VCC), 7 (GPIO -> In1) and 9 (Gnd -> Gnd) are needed
  • attach the relay between the another power cable going to the electric lock

Stage 3 (software):

Reader uses SPI, so enable SPI at raspi-config. TFT will rotate by adding display_rotate=1 (90 degrees) or display_rotate=3 (270 degrees) to /boot/config.txt – and console font can be made bigger with dpkg-reconfigure console-setup.

For actual use you need a small stack of software. Relay module must be controller (to open/close door) and NFC-card must be read and validated against some user database. And unless you want to keep the user database up-to-date by hand, there should be something for that also…

My solution was a simple python-service, which reads the NFC-card, validates is against data in MariaDB and after that tells the result on screen & activates the relay if necessary. Almost up to date code can be found at https://macronet.fi/dev/nfc/ (maybe later at github also) – it might work, or not.

Library used for reading the NFC-card is nxppy -that and all other requirements are easy to install:

# sudo apt install python-dev python-pip python-mysqldb cmake mariadb-server-10.0
# pip install nxppy

Stage 4 (installation):

Install the system to the place you want, so that you can show a card to the reader. This might need some cables if you want to take the reader further away – please make note that the reader requires 14 wires (haven’t actually tested if it works without i2c, because it uses SPI) so it’s like 2 cat5e cables.

Stage 5 (why internet connection? and then what?)

Make RasPi to call home/VPN, so that you are able to fix problems remotely.

Or “forget” a wireless keyboard receiver to USB-port and use your new NFC-enabled access control system as an IRC-client.

Jul 012017
 
English version here.

Miksi?

Tulipa eräänä päivänä tarvetta toteuttaa kulunvalvonta ja varauskalenteri erääseen yhteiskäytössä olevaan saunatilaan, missä oli ovenkarmissa jo sähkövastinrauta (24VAC) valmiina.

Käytetty rauta:

Vaihe 1 (muistikortti):

Raspberry Pi  sai sisäänsä uusimman Raspbian Jessien, joka tekohetkellä oli versiota 2017-06-21. Jakelun kernelinä toimii 4.9.28-v7+.

Vaihe 2 (kokoonpano):

  • irrota TFT:stä ohjainkortti (valitsemani kotelo vaati tämän)
  • kokoa TFT:n kotelo näytön ympärille
  • kiinnitä TFT:n ohjainkortti takaisin paikoilleen, varmista että molemmat lattakaapelit ovat hyvin kiinni
  • poraa raspin kotelon pohjaosaan oikeisiin kohtiin reiät (6mm terällä) TFT:n kiinnikkeitä varten
  • laita muistikortti raspiin
  • kiinnitä raspi koteloineen TFT:n kiinnikkeisiin
  • kiinnitä näytön lattakaapeli sekä virtajohdot 5V(pin4) ja Gnd(pin6) hyppylangoilla raspiin
  • kiinnitä NFC-lukija hyppylangoilla raspiin, tarvitset pinnejä 1,3,5,11,12,15,16,19,21-26 (mm. 3.3V, Gnd, i2c & SPI)
    (Explore NFC Board Schematic)
  • kiinnitä relekortti raspiin, rele tarvitsee toimiakseen ~80mAh eli huomioi tämä virtalähteen mitoituksessa
    tarvitset pinnit 2 (5V -> VCC), 7 (GPIO -> In1) ja 9 (Gnd -> Gnd)
  • kierrätä sähkölukon toinen virtajohto releen kautta

Vaihe 3 (softa):

Lukija käyttää SPI:tä, eli enabloi SPI raspi-config :sta. TFT käännetään pystyyn lisäämällä /boot/config.txt :n display_rotate=1 (90 astetta) tai display_rotate=3 (270 astetta) ja konsolin fonttia saa kasvatettua dpkg-reconfigure console-setup :lla.

Varsinaista käyttöä varten tarvitaan pieni nippu sovelluksia. Pitää esimerkiksi hallita relekorttia (avata/sulkea ovi) sekä lukea nfc-kortteja ja validoida luettu kortti jotain käyttäjäkantaa vasten. Ja ellei halua täyttää käyttäjäkantaa käsin niin niiden ylläpitoa vartenkin on hyvä olla omat sovelluksensa…

Oma ratkaisu oli toistaiseksi yksinkertainen python-service joka lukee nfc-kortin, validoi sen MariaDB:tä vasten jonka jälkeen kerrotaan ruudulla tulos & aktivoidaan rele tarvittaessa. Lähes ajantasalla olevaa koodia löytyy osoitteesta https://macronet.fi/dev/nfc/ (ehkä joskus myös githubissa) – se voi toimia tai olla toimimatta.

NFC-kortin luentaan käytetty nxppy -kirjasto ja muut tarvittavat riippuvuudet on helppo asentaa:

# sudo apt install python-dev python-pip python-mysqldb cmake mariadb-server-10.0
# pip install nxppy

Vaihe 4 (paikalleenasennus):

Asenna järjestelmä haluamaasi paikkaan paikoilleen niin että nfc-lukijalle voi näyttää korttia. Tämä voi vaatia hieman kaapelointia jos haluat viedä lukijan kauemmaksi raspista – huomioithan että lukija kaipaa 14 kaapelia (ei tosin ole tullut testattua _vaatiiko_ se niitä oikeasti, vai voiko esim. i2c:t jättää kytkemättä koska käyttää SPI:tä) eli 2kpl cat-kaapeleita.

Vaihe 5 (miksi internet? mitä sitten?)

Laita raspi soittamaan kotiin/VPN:ään, että pystyt korjaamaan mahdolliset vikatilanteet etänä.

Tai “unohda” usb-porttiin langattoman näppiksen vastaanotin ja käytä uutta nfc-kulunhallintajärjestelmääsi vaikka irkkaamiseen lokaalisti.

Feb 202016
 

I had a classic problem, old server without a remote KVM capabilities – if it goes down, no other chances to debug it than going next to it, plug in a monitor and a keyboard.

Then, one day, an epiphany. There is a “Console redirection” in BIOS, and after a quick enable & check – it really worked. BIOS was accessible over serial connection.

Next step, /etc/default/grub (under Debian) open and GRUB_CMDLINE_LINUX="" -> GRUB_CMDLINE_LINUX="console=ttyS0 console=tty0" will enable Linux console through serial. Switching from #GRUB_TERMINAL=console -> GRUB_TERMINAL="console serial" enables Grub to work with serial.

A quick update-grub made the changes active, and during reboot it was confirmed that the whole boot from bios screen to linux login console was accessible through serial connection.

 

So what about Raspberry Pi?

setupBy default Pi has a serial console active, which can be accessed through GPIO pins 8 & 10 (Pi B+). This needs to be disabled to make console port usable for userspace, by removing console=ttyAMA0 from /boot/cmdline.txt (or by disabling a correct service). After spending less than 1€ on Ebay (to get “MAX3232 RS232 Serial Port To TTL Converter Module DB9 Connector With Cable”) I had the required hardware to connect them together.

– Fits in 1U?  *** Yes

– Pretty setup?  *** No

– Works?  *** Yes

 

 

 

serialPi was powered via an USB port on the server backplane, a CAT5e -cable was needed for a network connection and the RS232 -converter was plugged into COM1.

Setup through the COM2 header found in the corner of the motherboard would have been nicer, but it just refused to work.

There’s also cables from the Pi into the reset button -pins on the motherboard, because I needed a way to make a hard reset. Power On/Off could also be managed through power button-pins, but for me it’s quite useless because the Pi is powered by the server through USB (and the server is always on).

 

Now I can access the Pi through SSH-session and use it to access console on the main server, if I ever need to do so.

 

GPIO connections (Pi B+ -> other end):
Pin 2 -> RS232 converter: VCC
Pin 6 -> RS232 converter: GND
Pin 8 -> RS232 converter: TXD
Pin 10 -> RS232 converter: RXD
Pin 14 -> Motherboard: Reset GND
Pin 16 -> Motherboard: Reset +

Reset-software (python-rpi.gpio required, don’t blame me if it destroys everything):

#!/bin/python
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BOARD)
GPIO.setup(16, GPIO.OUT)
# We're currently sinking the current through Pi.
# This causes a "push event" on a reset button.
GPIO.setup(16, GPIO.IN)

Jul 292015
 

Just a quick CPU benchmark with Phoronix test suite, comparing -cpu qemu64 and -cpu host (on Xeon E3-1241v3, 4vCPU/4GB virtuals).

Note: single run (fire & forget), virtual machines on the same physical hardware and benchmarks run simultaneously. YMMV.

Benchmark qemu64 host Higher/Lower better?
pts/stream-1.2.0 [Type: Copy] 14166.98 MB/s 14171.50 MB/s Higher
pts/stream-1.2.0 [Type: Scale] 13964.65 MB/s 14007.75 MB/s Higher
pts/stream-1.2.0 [Type: Triad] 15755.66 MB/s 15851.89 MB/s Higher
pts/stream-1.2.0 [Type: Add] 15730.77 MB/s 15863.64 MB/s Higher
pts/apache-1.6.1 25028.87 Requests Per Second 35859.45 Requests Per Second Higher
pts/john-the-ripper-1.5.1 [Traditional DES] 5849000 Real C/S 7179500 Real C/S Higher
pts/john-the-ripper-1.5.1 [Blowfish] 3170 Real C/S 3249 Real C/S Higher
pts/ttsiod-renderer-1.5.0 102.00 FPS 93.75 FPS Higher
pts/x264-1.9.0 86.56 FPS 94.52 FPS Higher
pts/graphics-magick-1.6.1 [HWB Color Space] 173 Iterations Per Minute 170 Iterations Per Minute Higher
pts/graphics-magick-1.6.1 [Local Adaptive Thresholding] 92 Iterations Per Minute 88 Iterations Per Minute Higher
pts/graphics-magick-1.6.1 [Sharpen] 95 Iterations Per Minute 102 Iterations Per Minute Higher
pts/graphics-magick-1.6.1 [Resizing] 159 Iterations Per Minute 156 Iterations Per Minute Higher
pts/himeno-1.1.0 1722.79 MFLOPS 1840.71 MFLOPS Higher
pts/compress-7zip-1.6.0 10173 MIPS 10291 MIPS Higher
pts/c-ray-1.1.0 44.54 Seconds 39.76 Seconds Higher
pts/compress-pbzip2-1.4.0 11.99 Seconds 13.95 Seconds Lower
pts/smallpt-1.0.1 153 Seconds 153 Seconds Lower
pts/crafty-1.3.0 71.60 Seconds 71.48 Seconds Lower
pts/encode-flac-1.5.0 7.38 Seconds 6.12 Seconds Lower
pts/encode-mp3-1.4.0 11.71 Seconds 11.21 Seconds Lower
pts/ffmpeg-2.4.0 18.99 Seconds 14.37 Seconds Lower
pts/povray-1.1.2 310.35 Seconds 248.59 Seconds Lower
pts/tachyon-1.1.1 26.57 Seconds 17.10 Seconds Lower
pts/openssl-1.9.0 286.50 Signs Per Second 543.67 Signs Per Second Lower
pts/mafft-1.4.0 7.40 Seconds 7.24 Seconds Lower
pts/gcrypt-1.0.3 1793 Microseconds 1647 Microseconds Lower

No wonder why UpCloud changed their parameters, -host usually wins…

Jun 222015
 

If you get hit with “RTNETLINK answers: Cannot allocate memory” when trying to add IPv6 default gateway back after losing all IPv6 -connectivity, raise net.ipv6.route.max_size.

The defaults (on my machines) were quite small compared to IPv4:

CentOS 7.0:
# sysctl net.ipv4.route.max_size
net.ipv4.route.max_size = 2147483647
# sysctl net.ipv6.route.max_size
net.ipv6.route.max_size = 4096

CentOS 7.1:
# sysctl net.ipv4.route.max_size
net.ipv4.route.max_size = 2147483647
# sysctl net.ipv6.route.max_size
net.ipv6.route.max_size = 16384

Debian 7.8/8.1:
# sysctl net.ipv4.route.max_size
net.ipv4.route.max_size = 2147483647
# sysctl net.ipv6.route.max_size
net.ipv6.route.max_size = 4096

Jun 022015
 

Finnish ISP DNA revealed their IPv6 -support for cable networks, but just listed “these devices are compatible” without more technical details.

How to get running (I used OpenWRT and EPC3825 was just a bridge):
* use DHCPv6-client
* request an /56 -prefix
* router will get an /128 -address and I found my /56 -network at the routes (with ip -6 route)
* drop a slice from it (/64) with radvd to your lan side
* remember to configure an IPv6 -firewall
* enjoy

Apr 012015
 

Beware, so you (too) won’t be bitten by this.

If you’re running vanilla KVM with default qemu-kvm -packages in RHEL (CentOS and probably others):

https://git.centos.org/blob/rpms!qemu-kvm!/958e1b8dccf9809600478f316ab641d490881fe7/SOURCES!kvm-rhel-Drop-machine-type-pc-q35-rhel7.0.0.patch;jsessionid=mc5igseasal0axkra6qhd4it

In short, q35 -support was a technical preview which is now dropped from qemu-kvm (qemu-kvm-rhev -only feature from now on), back to “pc”…

Oct 142014
 

Usually one of the first things we want to do with a new server is to restrict access to SSH -service.

So far it seems that everyone advices “disable firewallD, install iptables service and use it like you’ve always used” but how about trying to get along with this new tech?

Restricting access to SSH isn’t as hard as it might seem at the first glance. First we check what services are allowed in public (usually the default) and internal -zones:

# firewall-cmd --zone=internal --list-services
dhcpv6-client ipp-client mdns samba-client ssh
# firewall-cmd --zone=public --list-services
dhcpv6-client ssh

Then we add our admin-IP to internal -zone:

# firewall-cmd --permanent --zone=internal --add-source=<admin-ip>

Remove access to SSH-service from public:

# firewall-cmd --permanent --zone=public --remove-service=ssh

And reload the changes into use:

# firewall-cmd --reload

–permanent makes changes which stay over reboot/reload, but they aren’t active immediately – without –permanent the changes are active immediately but are lost on reload/reboot

Service definitions can be found (in RHEL/CentOS 7) at /etc/firewalld/services/ – if you create a new one -> use –reload to make it active.